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ABSTRACT 


The U.S. Army Training and Doctrine Command (TRADOC) Analysis Center’s 
Joint Dynamic Allocation of Fires and Sensors (JDAFS) model, a low-resolution, 
Discrete Event Simulation Model with embedded optimization enables the analysis of 
many scenarios and factors to explore Joint Intelligence, Surveillance, and 
Reconnaissance (ISR) missions. JDAFS is a powerful model that combines both discrete 
event simulation and the optimization of a linear objective function to generate realistic, 
reasonable, and consistent solutions to difficult ISR scheduling problems. Given a 
scenario and a mix of ISR platforms, JDAFS optimizes a flight schedule and executes the 
missions. This research develops a Joint ISR scenario, explores scenario simulation 
results, and provides a proof-of-principle analysis that aids in the ISR decision making 
process. 

This study examines 274 design points in each of two scenarios, a non-penetrating 
scenario that allows only standoff collection and a penetrating scenario that allows 
country of interest overflight. The use of an efficient design of experiment methodology 
enables the exploration of the interior and exterior of the response surface for the two 
experimental scenarios. Analysis of the simulation output suggests that the optimization 
interval significantly impacts total coverage. In the non-penetrating scenario, shorter 
optimization intervals ensure better coverage; however, in the penetrating scenario, 
longer optimization intervals provide for improved coverage. The disparity is explained 
by reduced likelihood of assignment saturation in the penetrating scenario due to the 
increased number of mission areas. Sensor range, sensor package configuration, and 
platform dwell time also affect the level of coverage. This is clearly demonstrated by the 


superior coverage provided by the most capable ISR platforms. 
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DISCLAIMER 


The reader is cautioned that the computer programs presented in this research may 
not have been exercised for all cases of interest. While every effort has been made, 
within the time available, to ensure that the programs are free of computational and 
logical errors, they cannot be considered validated. Any application of these programs 


without additional verification is at the risk of the user. 
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EXECUTIVE SUMMARY 


A well coordinated and _ synchronized Intelligence, Surveillance, and 
Reconnaissance (ISR) program is essential to maintaining global situational awareness 
and when necessary enabling a response with decisive force. The platforms that conduct 
ISR missions are a diverse collection of low density, high demand assets with complex 
operating requirements. These aircraft must be operated in a variety of environments and 
are subject to limitations in effective range and deployment locations. To further 
complicate matters, the ability of sensors to collect on intelligence targets of interest is 
not unlimited. When the targets are beyond the range of the collection platforms and 
sensors, it is necessary to penetrate the airspace of sovereign nations to gather the 
required intelligence. This is typically only done during a state of conflict and inherently 
brings considerable risks. This research develops a Joint ISR scenario, explores scenario 
simulation results, and provides a proof-of-principle analysis that aids in the ISR decision 


making process to give the responsive edge to Combatant Commanders. 


The U.S. Army Training and Doctrine Command (TRADOC) Analysis Center’s 
Joint Dynamic Allocation of Fires and Sensors (JDAFS) model, a low-resolution, 
Discrete Event Simulation Model with embedded optimization enables the analysis of 
many scenarios and factors. This thesis identifies critical JOAFS improvements for the 
representation of these Joint missions and explores the effects of 21 factors on the 
Measures of Effectiveness provide by the model. This research examines: 


1. What are the trade-offs associated with penetrating versus non-penetrating 
ISR missions? 


2: What are the marginal effects of increasing or decreasing airborne assets 
within a scenario? 


3: What are the appropriate design of experiment (DOE) methodologies and 
tools for penetrating versus non-penetrating ISR mission analysis? 


4, What are the improvements to the JDAFS simulation needed to credibly 
represent the Joint ISR mission? 


JDAFS is a powerful model that combines both discrete event simulation and the 


optimization of a linear objective function to generate realistic, reasonable, and consistent 
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solutions to difficult ISR scheduling problems. Given a scenario and a mix of ISR 
platforms, JDAFS optimizes a flight schedule and executes the missions. JDAFS 
considers the airframe operational parameters, sensor payload capabilities, base and 
mission area locations, and line-of-sight inputs in the solution to the ISR scheduling 


problem. 


The scenarios built for this study are designed to conduct a trade-off analysis for 
Joint ISR operations. Two types of scenarios were generated for examination and 
comparison within JDAFS. The first scenario is a non-penetrating scenario where the 
ISR platforms do not penetrate the Country of Interest’s (COI) national airspace. The 
internationally accepted buffer of 22 kilometers is respected on all flights and waypoints 
have been implemented to prevent ingressing and egressing aircraft from violating the 
COIs sovereign airspace. The second scenario assumes that conditions have changed to 
allow the violation of the COI’s airspace. With the incursions into the COI’s territory 
comes the risk of engagement by air defense assets, in this case surface-to-air missiles 
(SAMs). Figure | illustrates the penetrating scenario designed for this experiment. The 
only difference between the penetrating scenario and the non-penetrating scenario is the 


inclusion of mission areas within the border of the COI for the penetrating scenario. 


The country of interest for the scenario is depicted as a hexagon measuring 
1000km between any of its widest points. The size of the country allows the missions 
(ISR targets) to be widely dispersed and ensures that even the most capable sensors 
included in the study do not cover excessive portions of the country from a single mission 
area. The missions are dispersed non-uniformly throughout the country to achieve a 
sense of realism. While the mission dispersal is not uniform, it is also not random. Care 
is taken to place the missions such that the various mission sensor requirements are 
widely distributed to limit disproportionate coverage by a single platform. Mission areas 
were initially designed to ring the entire COI; however, allowing complete access to all 
countries surrounding a COI may be unrealistic. Overflight of mission areas along the 


western border and a portion of the southern border were deemed to be denied. Four 
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operating bases for the ISR platforms were placed at varying distances in the non-denied 
areas surrounding the COI to allow for different transit times from base to mission area. 


The air defense sites were distributed throughout the COI. 


TARGET (MISSION) 
BASE 


ag) SAM SITE 


MISSION AREA 
22km AIRSPACE BUFFER 
=== COUNTRY BORDER 





Figure 1. | JOAFS Scenario 


This study implements an experimental design technique known as the Nearly 
Orthogonal Latin Hypercube (NOLH) to explore how changes in the input parameters 
(factors) affect the model output. The 21 factors to be varied in this experiment consist 
of 20 numbers of aircraft at a base, and 1 optimization interval factor. The 20 aircraft 
factors take on integer values between O and 3 or 0 and 6, depending on the type of 
airframe. The optimization interval factor is a continuous variable with values between 


0.5 and 6 hours. 


The NOLH design provides a number of highly desirable qualities, including 
space filling, orthogonality, and flexibility. Space filling designs provide a means to 
explore the interior and exterior regions of the response surface, allowing a thorough 


examination of the output data without the need to run every conceivable design point. 
XXi 


Orthogonality ensures independence of the regressors, or coefficient estimates in 
resultant regression models. The NOLH design is also flexible, allowing rapid changes 
to input factors, factor levels, and the size of the overall design space. A sophisticated 
NOLH design template was used in the creation of the experimental design for this 


research. 


The final design for this experiment includes 274 design points to explore two 
scenarios; a non-penetrating scenario that allows only standoff collection and a 
penetrating scenario that allows country of interest overflight. The ISR platforms are 
subject to attrition by air defense assets in the penetrating scenario but not in the non- 
penetrating scenario. The scenarios each include 25 collection targets broken down into 
55 individual sensor requirements. An analysis of the output data from the two scenarios 
executed by the JDAFS model suggest: 


e Optimization is the most significant factor in both the scenarios, but has 
the opposite effect. In the non-penetrating scenario, shorter optimization 
intervals are better, but in the penetrating scenario longer optimization 
intervals yield improved coverage. Increased assignment options in the 
penetrating scenario causes this disparate behavior by the optimization 
interval factor. 


e Coverage is not necessarily improved by adding more ISR assets in the 
non-penetrating scenario. 


e Coverage is improved by the addition of assets in the penetrating scenario. 
Coverage would likely eventually plateau with diminishing returns from 
the addition of more aircraft in this scenario. The maximum number of 
airframes in the scenario was not sufficient to explore this behavior. 


e Penetrating ISR platforms average 20% more mean coverage than non- 
penetrating platforms. 


e The U-2 is the most important platform factor in explaining the variability 
in both scenarios. 


e The RQ-4 results in the majority of mission coverage for both scenarios 
due to its ability to remain on station. 


NOTE: The ISR platforms and sensors used in this study and _ their 
operational parameters and capabilities DO NOT represent actual classified 
real world performance characteristics. 
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In addition to exploring the output from the simulation, JDAFS was exercised to 


evaluate additional changes that would be required to better model Joint ISR missions. 


Several recommended additions or improvements were identified: 


Allow ISR platforms to receive credit for collection on targets within 
sensor range during ingress and egress to mission areas. 


Create probability distributions to represent current and future force 
sensors from certified data to accurately represent the acquisition of 
targets by sensors in the model. 


Implement revisit valuation schemes for missions to more credibly 
represent ISR mission characteristics. 


Addition of joint assets (platforms, sensors, munitions, etc.) in the JDAFS 
database for reuse in joint studies. 


Further develop the functionality of JDAFS to more credibly represent 
Joint Sensors to include satellite assets. 


Provide a more robust representation of unmanned aerial vehicles with 
weapons and sensors that allow force-on-force analysis of differing 
platform mixes. 


Develop an integrated design of experiments interface that enables a quick 
determination of alternatives. 


Incorporate the InputGenerator and OutputGenerator scripts into the 
JDAFS program. The integration of these tools will eliminate the need for 
the pre and post-processing of data and eliminate the need to transfer 
excessively large files. 


Develop a simple, well documented JDAFS interface to allow an analyst 
to easily enter the data for the various input factors. 


Develop a comprehensive JDAFS Users/Analyst Manual. 
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I. INTRODUCTION 


A. BACKGROUND 


The national security of the United States faces dynamic challenges across the 
globe. A well coordinated and synchronized Intelligence, Surveillance, and 
Reconnaissance (ISR) program is essential to maintaining global situational awareness 
and when necessary enabling a response with decisive force. “We know that if the 
intelligence cycle lags behind operational and command decision-making windows it is 
not actionable. Just as a maneuver formation on the ground, at sea or in the air is a 
warfighting system, so is intelligence. By making ISR the responsibility of a functional 
Combatant Commander we gain synergy and perspective among warfighting systems” 
(U.S. Congress 2007, 7). The outcome of this research provides tools that aid in the ISR 
decision making process for the USSTRATCOM Global Defense Posture and give the 


responsive edge to Combatant Commanders. 


The need to plan and act globally is more important than ever before. Writing on 
the shift from a regional to a global perspective, General Myers (2003), as Chairman of 
the Joint Chiefs of Staff, noted that, 


We will need global ISR activities for gathering indications and warning 
data and for otherwise enabling global strike, space operations, certain 
elements of IO, and integrated missile defense. Moreover, we need global 
C2 capabilities to enable integrated global missile defense, facilitate global 
strike, integrate regional operations with global operations, and integrate 
regional operations in one AOR with those of another. Knitting together 
various regionally focused ISR activities is unlikely to yield a coherent 
global perspective. Simply put, we cannot obtain a relevant global 
perspective without ISR activities that, to some degree, are globally 
coordinated and directed—a function performed by the Defense 
Intelligence Agency. The new factor is that, given the low-density/high- 
demand nature of many of our ISR resources, regional combatant 
commands are more likely than before to be required to conduct ISR 
activities in support of global operations. 


Reflecting the shift in perspective, Change Two to the Unified Command Plan 
designated US Strategic Command (USSTRATCOM) as the executive agent for the 
Global ISR mission.!_ The assignment of the Global ISR mission to USSTRATCOM 
ensures that a comprehensive, global approach is applied to provide timely, survivable, 


persistent coverage of key targets or areas of interest. 


The challenge of meeting the nation’s ISR demands is enormous. The platforms 
that conduct ISR missions are a diverse collection of low density, high demand assets 
with complex operating requirements. These aircraft must be operated in a variety of 
environments and are subject to limitations in effective range and deployment locations. 
To further complicate matters, the ability of sensors to collect on intelligence targets of 
interest is not unlimited. When the targets are beyond the range of the collection 
platforms and sensors, it is necessary to penetrate the airspace of sovereign nations to 
gather the required intelligence. This is typically only done during a state of conflict and 


inherently brings considerable risks. 


A need exists to evaluate the trade-offs based on various ISR platform capabilities 
and basing configurations. The Joint Dynamic Allocation of Fires and Sensors (JDAFS) 
model, a low-resolution, Discrete Event Simulation Model with embedded optimization 
enables the analysis of many scenarios and factors in a relatively short time period when 
compared to other simulations (U.S. Army Training and Doctrine Command 2007a). 

This research is sponsored by the U.S. Army Training and Doctrine Command 
(TRADOC) Analysis Center (TRAC), Monterey. Collocated with the Naval 
Postgraduate School, TRAC-Monterey is focused on research in computer simulation 
technology and military operations research. “TRAC Monterey is recognized as a 
premier applied research organization for military modeling, simulation, methodologies, 


and analysis” (U.S. Army Training and Doctrine Command 2007b). 





! President Bush signed Change Two to the Unified Command Plan on Jan. 10, 2003, and tasked 
USSTRATCOM with four previously unassigned responsibilities: global strike, missile defense integration, 
Department of Defense Information Operations, and C4ISR (command and control, communications, 
computers, intelligence, surveillance, and reconnaissance) (U.S. Strategic Command, 2007). 
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B. PURPOSE OF RESEARCH 


The purpose of this research is to simulate, by use of the JDAFS model, the 
effects of varying key factors associated with the operation and employment of manned 
and unmanned ISR platforms. The main focus of this research is an analysis of the trade- 
offs associated with specific platforms and basing decisions on the execution of 
penetrating vs. non-penetrating missions. The analytical findings provide a foundation 
for future ISR scheduling and requirements decisions. Additionally, prior to 
commencing production runs, the JDAFS model will be exercised to identify any 
problems or deficiencies that would prevent the use of this model for the planned 


research. 


Cc: RESEARCH QUESTIONS 


The flexibility and power of the JOAFS model to simulate dynamic ISR missions 
offers a number of areas for potential research. This research will explore: 


1. What are the trade-offs associated with penetrating versus non-penetrating 
ISR missions? 


2: What are the marginal effects of increasing or decreasing airborne assets 
with in a scenario? 


a What are the appropriate design of experiment (DOE) methodologies and 
tools for penetrating versus non-penetrating ISR mission analysis? 


4, What are the improvements to the JDAFS simulation needed to credibly 
represent the Joint ISR mission? 


D. SCOPE OF RESEARCH 


The Design-of-Experiments (DOE) and JDAFS simulation are developed based 
on two fictional, yet realistic scenarios using data acquired from unclassified sources. In 
establishing the DOE, specific factors that contribute to the ISR platform attributes of 
timely, survivable, and persistent are identified. The identified factors are examined at 
various ranges within the DOE framework. However, the number of factors involved 


within a full-factorial design makes examining every combination of inputs impractical. 


Therefore, an efficient, robust DOE is employed to examine the response surface. Output 
from the simulation runs is analyzed to evaluate how the various input factors affect the 


predetermined Measures-of-Effectiveness (MOE) across the range of scenarios. 


E. THESIS ORGANIZATION 


Chapter II provides a history and description of the Joint Dynamic Allocation of 
Fires and Sensors (JDAFS) model. Chapter III is an overview of the scenarios to be 
executed along with a listing of the various manned and unmanned aircraft and sensors 
modeled in the simulation. Chapter IV covers the design of experiments to facilitate the 
running of the JOAFS model. Chapter V details the analysis completed on the model 
output data. Chapter VI is a summary of findings based on the research conducted and 


includes recommendations for follow-on study. 


Yl. JDAFS MODEL, INPUTS AND METHODOLOGY 


A. HISTORY OF THE JDAFS MODEL 


The Dynamic Allocation of Fires and Sensors (DAFS) model was created for 
TRAC-Monterey in 2002 (Havens 2002). DAFS is a low-resolution, constructive entity- 
level simulation framework programmed in JAVA and incorporates the functionality of 
the Simkit simulation toolkit created by Dr. Arnold Buss (2006). The original version of 
the model provided a low-resolution simulation tool for TRAC-Monterey to explore 
scenarios for the Army’s Future Combat System. JDAFS has undergone substantial 
revision and expansion by TRAC-Monterey to improve the functionality and usability of 
the model as an analysis tool. In early 2007, the model name was changed to Joint 
Dynamic Allocation of Fires and Sensors (JDAFS) in recognition that the model’s 


functionality had expanded beyond use for purely Army studies. 


Between its original creation in 2002 and early 2007, DAFS has evolved into two 
separate models: 


e Joint Dynamic Allocation of Fires and Sensors (SDAFS), which consisted 
primarily of the simulation and visualization portion of the program 


e Assignment Scheduling Capability for Unmanned Aerial Vehicles (ASC- 
U) tool, which retained the scheduling and optimization functionality. 


As part of this effort, the two programs have subsequently been recombined in the 


current version of JDAFS. 


B. JDAFS ORIGINAL CONFIGURATION 


ASC-U and JDAFS were developed from DAFS but were two separate software 
development branches. ASC-U provided a robust scheduling capability for manned and 
unmanned aerial sensor platforms using optimization to make assignments using an 
approximate dynamic programming approach. ASC-U was deterministic since no 
stochastic effects were considered by the methodology. DAFS provided a low-resolution 
approach to force-on-force combat and relied on its optimization capability for fires 


assignment. DAFS represented the stochastic effects of the fires actions. 


if JDAFS Functionality 


JDAFS, in its original configuration, was an entity level simulation that relied on 
scripted scenarios to execute a simulation. Fires allocation was conducted throughout the 
scenario via an optimization algorithm based on matching available weapons and firing 
platforms to targets, dependent on acquisition sensor input. The adjudication of fires was 
a stochastic process driven by a pre-determined distribution. Once a simulation 
commenced, the scenario would run to completion as scripted with no ability for dynamic 


re-tasking. 


Z; Event Graph Methodology 


The DAFS discrete event simulation model is built on the concept of Schruben’s 
(1983) event graph methodology. The event graph approach is uniquely suited to 
describe and implement the types of movement, sensing, and weapons effects interactions 
required in a low-resolution simulation. The use of event graph methodology can provide 


significant advantages over a time stepped approach with respect to model run time. 


The foundation of event graph methodology is the event list. Discrete event 
simulations are run by sequentially executing events on the event list. Buss and Ahner 


(2006b, 1353) provide the following explanation of the event graph: 


An Event Graph consists of nodes and directed arcs. Each node 
corresponds to an event, or state transition, and each arc corresponds to the 
scheduling of other events. Each arc can optionally have an associated 
boolean condition and/or a time delay. Figure 2 shows the fundamental 
construct for Event Graphs and is interpreted as follows: the occurrence of 
Event A causes Event B to be scheduled after a time delay of ¢, providing 
condition (i) is true (after the state transitions for Event A have been 
performed). By convention, the time delay t is indicated toward the tail of 
the scheduling edge and the edge condition is shown just above the wavy 
line through the middle of the edge. If there is no time delay, then t is 
omitted. Similarly, if Event B is always scheduled following the 
occurrence of Event B, then the edge condition is omitted, and the edge is 
called an unconditional edge. Thus, the basic Event Graph paradigm 
contains only two elements: the event node and the scheduling edge with 
two options on the edges (time delay and edge condition). 





Figure 2. Event Graph Example (From Buss and Ahner 2006b) 


For additional information on event graph mythology as it applies to modeling movement 
and detection, see Simple Movement and Detection in Discrete Event Simulation by Buss 


and Sanchez (2005). 


Cy ASC-U ORIGINAL CONFIGURATION 


The scheduling of ISR platforms is an inherently dynamic and complex process. 
The standalone ASC-U program is a powerful scheduling tool that combines both 
simulation and optimization to help overcome these complexities. Provided with high 
quality input data, ASC-U is capable of generating realistic, reasonable, and consistent 


solutions to difficult ISR scheduling scenarios. (Ahner et al. 2006b, Nannini 2006) 


Despite using simulation to determine its solutions, ASC-U is not a stochastic 
model. It produces identical flight schedules for a given set of input parameters. The 
simulation portion of the model overcomes the discrete time step artificiality that can be 


introduced by multi-period optimizations (Ahner et al. 2006a). 


While ASC-U is primarily designed for the scheduling of UAV missions, it easily 
incorporates mixes of manned and unmanned aircraft to solve the following challenge 
(Ahner et al. 2006a, 2): 

Given a scenario that specifies the number of each type of UAV, initial 

UAV locations, and UAV performance characteristics, determine the 

number of missions that can successfully be completed and the schedule 

for each UAV. The solution must consider ground control station (GCS) 

locations and capacities, launch and recovery site (LRS) locations and 

capacities...and communication platform footprint and capacities. 


The ultimate schedule derived by ASC-U is driven by calculating the maximum total 


mission value based on the performance parameters of the available ISR platforms. 


1. Primary ASC-U Components 


ASC-U’s primary components consist of missions, ISR platforms, sensor 
packages, LRSs, GCSs. Each component is populated in a Microsoft Access database 


using one or more tables for the various elements. 


a. Missions 


Missions, or ISR targets, consist of a set of coordinates combined with a 
set of sensor requirements. For example, a surface-to-air missile site would have a set of 
locating coordinates and require coverage by electro-optical/infrared (EO/IR) sensors, 
synthetic aperture radar (SAR), and signals intelligence (SIGINT) sensors. In addition, 
each sensor requirement is assigned a value rate. ISR platforms receive credit or value by 
being on-station within sensor range of a sensor requirement provided that the mission, or 
sensor requirement, is not already being covered by another platform. The total value 
derived is found by multiplying the value rate of the sensor requirement by the ISR 
platform’s time on-station. ISR platforms that carry multiple sensor packages can fulfill 
multiple mission requirements. The value achieved by satisfying multiple mission 


requirements is additive (Ahner et al. 2006b). 


b. TSR Platforms 


Individual ISR platforms in ASC-U consist of a unique name, an 
associated LRS, a transition time, an airspeed, an operational endurance, a GCS operating 
radius, and one or more sensor types. Unique names automatically generate for the 
individual ISR platforms to allow mission assignments, the tracking or sensor 
requirements accomplished, and the state of the platform within the simulation. An ISR 
platform’s airspeed and operational endurance determine feasibility of assignment to 
specific missions and the amount of time the aircraft can remain on-station prior to 
returning to an LRS. A platform’s transition time accounts for the minimum time an 
aircraft must spend at an LRS for maintenance and refueling prior to launching for a new 
mission. The GCS operating radius? dictates the maximum distance a UAV ISR platform 


2 GCS Control Radius was assumed to infinite in this study. All communications with UAVs were 
deemed to occur via satellite and GCS control is not required for manned platforms. 
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can travel from a GCS without losing contact. Sensor types assigned to individual 
platforms determine the types of missions capable of being covered by specific aircraft 


(Ahner et al. 2006b, 1350). 


Cc. Sensor Packages 


Each ISR platform has an associated sensor package. These packages 
consist of sensor, weapon, or communications capabilities. These capabilities control the 


types of missions that are assignable to the various platforms (Ahner et al. 2006b, 1350). 


d. LRS 


LRSs provide an operating base from which the ISR platforms operate. 
An LRS has a unique name, an associated ISR platform type, a set of locating 
coordinates, and a capacity. While not used within the context of this thesis, LRSs have 


the ability to move if associated with UAVs (Ahner et al. 2006b, 1350). 


e. GCS 


The GCS provides a means for controlling UAV platforms during flight. 
Like an LRS, a GCS is composed of a unique name, an associated ISR platform type, a 
set of locating coordinates, and a capacity. Also, just as with LRSs, GCSs have the 


ability to move during a scenario (Ahner et al. 2006b, 1350). 


2. Simulation and Optimization 


Using the input data from the components described above, ASC-U uses the 
combination of optimization and simulation to deliver quality solutions to complex 
scheduling and assignment problems within an approximate dynamic programming 
framework. ASC-U is built upon the DAFS discrete event simulation model that contains 


an embedded constrained value optimizer (CVO). 


The CVO algorithm delivers a myopic optimal solution. The myopic, or locally 
optimal, solution is necessary to maintain problem tractability. The calculation of a 


globally optimal solution would become computationally infeasible for all but the 


smallest of problems. The CVO derives its solution by comparing the ISR resources 
available to the existing sensor requirements to determine the optimal scheduling to 


maximize total value (Ahner et al. 2006b, 1351). 


3. ASC-U Platform Assignment Process 


The following description from the ASC-U Users/Analyst Manual illustrates the 
ASC-U platform assignment process (Ahner et al. 2006a, 4-5): 


For example, consider 2 UAVs assigned to 4 missions as depicted in 
Figure 3. ASC-U performs optimal assignments at predetermined time 
intervals (1). At time to, assume two UAVs are available, UAV1 and UAV 
2. The tool considers all available UAVs with available GCS control and 
all missions with value in the fixed time horizon time window (missions 1, 
2, 3 but not 4 for time to) (2). A UAV is assigned to a mission and can 
service any sensor requirement associated with that mission if it has the 
correct sensor. Assume the optimal assignment is UAV 1 assigned to 
Mission | and UAV 2 assigned to Mission 2. UAV 1 is launched 
immediately to arrive at Mission | as soon as possible (3). UAV 2 does 
not launch but is scheduled for launch to arrive just-in-time for the 
beginning of Mission 2. At time t;, UAVs that are scheduled to launch but 
have not yet launched are “unassigned” and considered for assignment 
(UAV 2 is unassigned) (4). UAV 2 is again assigned to Mission 2 and 
scheduled for launch to arrive just-in-time. UAV assignments do not 
change once they have been launched. 


Mission 4 






Time window considered 
by optimization at t=t, 


———.—_——— 


Time window considered 
by optimization at t=t, 











Figure 3. | ASC-U Dynamic Cueing and Transitions (From: Ahner et al. 2006a) 
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The process ASC-U goes through in the construction of flight schedules is 
illustrated in Figure 4. Beginning in the Dynamic Simulation State Transitions box and 
proceeding clockwise, the system is prompted to execute an optimization. The Value of 
Potential Assignment Generator determines the potential value that could be accumulated 
within the allotted time horizon for all possible ISR platform assignments. The value 
derived at this stage accounts for all of the parameters affecting the aircraft, including 
travel time, time-on-station, and GCS capacity. The solution from the Value of Potential 
Assignment Generator is passed to the optimization routine. The optimization stage 
returns the locally maximal value of the maximization problem. This solution is passed 
to the UAV scheduler for platform assignment. The aircraft are slated to launch such that 
they arrive promptly at the mission start time, taking travel time into account. Following 
the preset re-optimization time interval the process reenters the Dynamic Simulation 
State Transition phase and the process continues until a predetermined termination time is 


met (Ahner et al. 2006a). 


The value of each assignment is determined BEFORE the optimization 


taking into account travel time, time-on-station, LRS location over time, 


and GCS location and capacity over time. 
Values of assignments are passed to Optimization. 






Optimization provides “optimal” 
assignment as cues for where 
airframes should go. 








Periodic or event triggered 
reoptimization 





The UAV Scheduler determines when airframes should 
be launched to locations given to it by the optimization. 


Figure 4. © ASC-U Dynamic Cueing and Transitions (From: Ahner et al. 2006a) 
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For additional details on installing and using the ASU-C tool, see the ASC-U 
Users/Analysts Manual. 


At every optimization event the CVO for the three-dimensional (3D) integer 
program (IP) must be solved. LP_Solve 5.5, programmed in C, is implemented in 
JDAFS and accessed using the Java Native Interface (JNI) (Ahner et al. 2006b, 1351). 
This solver algorithm is an implementation of the simplex method using the branch and 
bound technique to solve linear program and integer program formulations. The IP 


formulation in ASC-U is as follows: 


Indices: 
ie MP MissionPackage: A package with capabilities that is 
installed on a MissionPlatform when the MissionPlatform 
is launched. 
J<¢MA Mission Area: A location where a MissionPlatform with a 
MissionPackage patrols. 
k Ee Msn Mission: A location that requires coverage by a 
MissionPackage. 
le LRS LRS: A location that holds MissionPackages and 
MissionPlatforms that are not currently in use. 
g €GCS GCS: A ground control station. 
Sets: 
AvailPackage, The set of available MissionPackages at LRS | 
Areas, The set of MissionAreas assigned to GCS, 
(Assignment made by a heuristic prior to 
optimization) 
Data: 
ready _ platform, The number of MissionPlatforms ready for launch 
at LRS). 
capacity , The remaining GCS capacity at GCS,. 
costToLaunch The cost of assigning a MissionPackage to a MissionArea. 
Cin The value gained by assigning MissionPackage i to 


MissionArea j to cover Mission k. 
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Variables: 


x 


ik 


Formulation: 


1 


1 if MissionPackage i is assigned to MissionArea j to cover 
Mission k, 0 otherwise. 
1 if MissionPackage 7 is assigned to MissionArea j, 0 


otherwise. 


Max) Ci * X ig — costToLaunch ¥ | Y, 


ijk ij 


Subject to: 


2 


8. 
2: 


Explanation: 


ee a a 


Y<X, Vii 
k 


Xi SY, Vik 
ast Ni 

j 

DXi <1 Vk 
y 


> DY, < ready _ platform, ‘VI 


ieAvailPackage, j 


EY > Y,, Scapacity, Vg 


i jeArea, 


Xx. < (0,1) 


ijk 


Y, € (0.1) 


Maximize the total values of the assignments 
Force Y,,to be 0 if no Mission is covered by assigning MissionPackage i to 
MissionArea /. 


Force Y,,to be 1 if any MissionPackage i is assigned MissionArea /. 


Each MissionPackage is assigned to at most one MissionArea. 
Each Mission is covered by at most one MissionPackage. 


The number of MissionPackages from an LRS assigned does not exceed 
the number of MissionPlatforms ready for launch at the LRS. 


The number of MissionPackages assigned to MissionAreas covered by 
each GCS does not exceed the remaining GCS capacity. 


13 


8. Ensure that a MissionPlatform is assigned to only one MissionArea at a 
time. 


9. Ensure that a MissionPackage is assigned to only one MissionArea at a 
time. 


Cyx calculations: 


Cijx will be zero if MissionPackage i does not have the needed capability to cover 
Mission k or Mission k is not within range or line-of-sight of MissionPackage 7 
when at MissionArea j. Otherwise, Cj is the total value remaining for Mission k 
during the period that MissionPackage i can be on-station at MissionArea j, taking 
into account the endurance of the MissionPlatform carrying the MissionPackage 
and the ingress/egress time from the LRS at which the MissionPackage is located. 


Optimized schedules are ‘flown’ by the simulation portion of the DAFS model in 
order to capture the dynamics of the platform transitions and movement. The simulation 
functionality takes on increased importance in scenarios where fires and complex 


relationships between moving GCSs and LRSs are involved. 


D. CHANGES IMPLEMENTED IN JDAFS 


To maximize the usefulness of JDAFS as analytical tool, several changes are 
necessary. These improvements involve restoring some of the capabilities from the 
original model, as well as, adding new functionality. The following modifications are the 
initial changes identified to make the simulation viable for this study; further operational 
testing of JDAFS is necessary to identify any remaining bugs and deficiencies prior to 


conducting production runs. 


1; Scheduling Tool 


To simulate an ISR scenario from start to finish, some means of creating a flight 
schedule or Air Tasking Order (ATO) is required. While several options were 
considered, including building a fixed schedule prior to execution or the creation of a 
heuristic scheduling method, the best alternative was to integrate ASC-U back into 
JDAFS. This created not only the ability to build schedules on the fly, but also the means 
to optimize the flight schedule as the simulation progressed. In addition, ASC-U, 
coupled with pre-calculated line-of sight determinations between mission and mission 


areas solves a two stage assignment problem. 
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2. Mission Areas 


The creation of mission areas was required to provide a method to simulate an 
ISR platform on-station. As a low-resolution simulation, JOAFS does not model flight 
characteristics that would be included in a higher resolution model such as turning, 
climbing, descending, speed changes, and other more sophisticated performance 
parameters. In JDAFS, platforms actually transit from a base to a mission area and 
remain stationary at that mission area for the duration of the sortie. This artificiality is a 
trade-off between functionality in the form of minimizing solution times necessary for the 


execution of the model and realism. 


3. Bases 


The integration of ASC-U brings with it the ability to create Launch and 
Recovery Sites (LRS) and Ground Control Stations (GCS). Virtual operating bases can 
be established by setting a common set of coordinates for these entities and associating 
specific ISR platforms. ASC-U has the ability to account for mobile GCSs and LRSs, 
however, for this study, all GCSs and LRSs with identical coordinates are treated as fixed 


operating bases. 


E. IMPROVED JDAFS CONFIGURATION 


The current version of JDAFS is a more robust version of the original program. It 
incorporates all of the functionalities of the ASC-U program described above along with 
a richer simulation capability. The remainder of this section addresses specific details 


related to the description and execution of the improved JDAFS configuration. 


1. LOS Calculation 


The minimum altitude line-of-sight (LOS) between all missions and mission areas 
is pre-calculated prior to the execution of any JDAFS scenario. The predetermination of 
the minimum altitude LOS is what enables 3D optimization in the execution of JDAFS 
scenarios. For the purposes of this study, minimum altitude (LOS) is calculated in two 


steps. First, the distance between the mission and mission area is calculated as follows: 
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Distance = VG 


mission _ X mission area ) 7 ( Y mission ~ Y mission oe, 
Second, the minimum altitude required to achieve LOS is calculated?: 


(Distance) 
2 x 6378155 


Minimum Altitude = 


The results of the LOS calculations are then used to populate the MinLOSAIt field in the 
LOS input table. 





Figure 5. Minimum Altitude Line-of-Sight Determination 


The scenarios in this study assume flat terrain with only the curvature of the earth 
affecting the ability to achieve line-of-sight. The inclusion of manmade or terrestrial 
terrain features obstructing LOS are beyond the scope of this study. For the execution of 
JDAFS scenarios based on real world geographic locations, a LOS calculation tool has 
been created by TRAC-MTRY. The LOS tool uses Digital Terrain Elevation Data 
(DTED) to calculate the minimum altitude LOS based on the terrain at and around the 


actual coordinates of the missions and mission areas. 


ae Input Requirements 


JDAFS runs on an XML file created from inputs in an Access database. The 


Access database contains the various tables required for the execution of a scenario. 


3 The minimum altitude calculation is derived from the equation for determining the distance to the 
visual horizon: Distance to the visual horizon R = (2+ Rj, °/ , where h is the sensor height, in meters, 
and R,,,,, 18 the radius of the earth at the equator, 6378155 meters (Wittenberg 1997). 
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earth 


Only the data tables used for this study will be discussed further. For information on the 
complete list of tables and their uses see the ASC-U User’s/Analyst Manual (Ahner et al. 
2006b) and the JDAFS User’s Guide (Ahner et al. 2006c).*° 


a. DAFSScenario 


The DAFS Scenario table is the primary driver for the simulation. Key 
elements determined in this table include the optimization interval, the number of 
replications to be executed, and the length of the scenario to be simulated. 
DAFSScenario elements consist of: 


e Version: The version of the input file—used to determine the schema of 
the input. 


e Type: Currently only ‘Attack’ is used. 


e optimizeInterval: Time between optimization events for the CVO. 

e bdaF actor: A number between 0.0 and 1.0 representing the probability 
that BDA will be correct (not used). 

e replications: Number of replications to be done. 

° stopTime: Length of the scenario. 
b. GCS 


The Ground Control Station (GCS) table indicates the name, the 
associated LRS, and control capacity of each ground control station. A GCS and its 
associated LRS constitute an operating base for the launch, control, and recovery of ISR 
platforms. The GCS table elements are: 

e Name: A unique identifier for each GCS. 
e LRS: The name of the LRS associated with the GCS. 


e Capacity: The maximum number of platforms (i.e., UAVs) that the GCS 
can control at any one time. Only the first entry for each GCS is used. 





4 Data table names, letter case and spelling are taken directly from the current version of the JDAFS 
input tables. 


5 Descriptions of the elements in the data tables are taken from either the ASC-U Users/Analysts 
Manual or the JOAFS Users Manual as applicable. Where necessary, the wording has been changed to 
match the current usage in JDAFS. 
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Cc. GCSLocation 


GCS Location is associated with the GCS table to provide the geographic 


location of the GCS and the time at which it becomes available. For this study the GCSs 


are available throughout the entire scenario and their location is fixed. The GCSLocation 


table is comprised of the following elements: 


GCS: Name of this GCS from the GCS table. 


Time: The H-hour at which this GCS is at the location designated by the 
LocationX and LocationY elements. 


LocationX: The x coordinate of this GCS at the time in the Time element. 


LocationY: The y coordinate of this GCS at the time in the Time element. 
d. LinearKillProbability 


The linear kill probability table provides the elements necessary to 


evaluate the outcome of a firing event. Each munition type is named and must be 


specifically matched with a target platform vulnerable to attack with the weapon. The 


minimum and maximum range for the munition type and the respective kill probabilities 


(P,) are also defined in this table. Elements of the LinearKillProbability table are: 


munitionType: The munition for which the kill probability is defined. 
This must be defined in a MunitionType element. 


platformType: The target platform for the LinearKillProbability 
definition. This must be defined in a PlatformType element. 


class: JAVA class that adjudicates the firing action. 


minRangePK: The probability of kill against a target platform at 
minimum range. 


maxRangePK: The probability of kill against a target platform at 
maximum range. 


minRange: Minimum range at which a target can be engaged. 


maxRange: Maximum range at which a target can be engaged. 
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e. LOS 


The line-of-sight (LOS) table provides the necessary data for the two stage 
assignment problem. The values are pre-calculated to determine the minimum altitude at 
which there is an unobstructed line of sight between a mission and a mission (target). 
The elements that make up the LOS table are: 


e Mission: The name of the mission from the Mission table that corresponds 
the Mission Area for which the minimum altitude to maintain line-of-sight 
is designated in the MinLOSAIt element. 


e MissionArea: The name of the Mission Area from the MissionArea table 
that corresponds the Mission for which the minimum altitude to maintain 
line-of-sight is designated in the MinLOSAIt element. 


° MinLOSAIt: The minimum altitude in meters required to maintain line- 
of-sight between the Mission Area and the Mission locations. 


f _LRS 


Launch and recovery sites (LRS) are named and the capacity is defined in 
the LRS table. A LRS and its associated GCS constitute an operating base for the launch, 
control, and recovery of ISR platforms. The LRS-GCS association is done in the GCS 
table. The LRS elements consist of: 

e Name: A unique identifier for each LRS. 


e Capacity: The maximum number of each ISR platform at the LRS. 
g. LRSLocation 


LRS Location is associated with the LRS table to provide the geographic 
location of the LRS and the time at which it becomes available. For this study the LRSs 
are available throughout the entire scenario and their location is fixed. The LRSLocation 
table is comprised of the following elements: 

e LRS: The name of the LRS from the LRS table. 


e Time: The H-hour at which this LRS is at the location designated by the 
LocationX and LocationY elements. 


e LocationX: The x coordinate of this LRS at the time in the Time element. 


® LocationY: The y coordinate of this LRS at the time in the Time element. 
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h. MetaData 


The metadata table drives how a design scenario will be run by JDAFS. 
Most significantly, the MetaData table defines the type of optimization algorithm to be 
implemented, the length of the simulation scenario, the time between optimizations, and 
the location of the origin for the scenario coordinate system. Elements in the MetaData 
table are: 
e InputFormatVersion: Reports the current version of the input format. 


e CVOType: Identifies CVO JAVA class to use. 


° ScenarioLength is the length of the scenario to be run in simulation 
hours. 

° OptimizationInterval is the amount of time in simulation hours between 
reallocation of the ISR platforms. 

e ReportInterval: Frequency of report intervals (not used). 

e DiscountFactor: Discount for time into the future (not used). 

e EarlyReturn: Heuristic (not used). 

e SecondaryAreas: Heuristic (not used). 

e AppendAreas: Heuristic (not used). 

e LocationType: Heuristic (not used). 

® OriginLat: Sets origin latitude for the scenario. 

® OriginLng: Sets origin longitude for the scenario. 

e CostToLaunch: (not used). 
i. Mission 


The targets for collection by the ISR platforms are called missions. Each 
mission is defined by a name, a location, the type of sensor required to collect on the 
mission, and the value the correct sensor achieves per hour for coverage of the mission. 


The Mission table elements consist of: 


e Name: Unique identifier for each mission (target). 

e LocationX: The x coordinate of the mission (target). 

° LocationY: The y coordinate of the mission (target). 

e StartTime: The time that the sensor requirement starts, usually in H-hours 
(not used). 


20 


e EndTime: The time that the sensor requirement ends, usually in H-hours 


(not used). 
e RequiredCapability: The sensor type required to detect this mission. 
e ValueRate: The rate at which an ISR platform with the correct sensor 


receives credit when the mission (target) is in the ISR platform’s sensor 
footprint (all values assigned as 1.0 for this study). 


J. MissionArea 


A mission area is a notional on-station location for the ISR platforms 


consisting of a name and a location. The elements in the MissionArea data table are: 


e Name: Unique identifier for each mission area (patrol location). 

° LocationX: The x coordinate of the mission area (patrol location). 

e LocationY: The y coordinate of the mission area (patrol location). 
k. MissionPackageLocation 


The mission package location indicates how many of the mission 
packages, defined in the MissionPackageType table, are located at each LRS (base). A 
new row is required for each mission package at a LRS. The MissionPackageLocation 
attributes are: 


e LRS: Unique identifier for the Launch and Recovery Site at which the 
MissionPackageType (ISR platform) is located. Each LRS can only be 
associated with one MissionPackageType (ISR Platform). Multiple LRSs 
can be associated with a _ single geographic location in_ the 
MissionPlatformLocation table using the UnitName element. 


e MissionPackageType: Name of the ISR platform associated with the 
LRS. 


e QTY: Number of MissionPacakgeTypes at the LRS. 
L. MissionPackageType 


The mission package type defines the sensor packages that ISR platforms 
may carry. Each package has a unique number followed by unique names of sensors that 
together make up the package. Each row of the MissionPackageType table represents 


one sensor package. The MissionPackageType elements are: 
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Name: Unique name of an ISR platform. 


CapabilityType: Type of sensor payload associated with the ISR platform 
in the Name element. If the ISR has more than one sensor payload, each 
sensor must be listed individually with the corresponding Name element. 
m. MissionPlatformLocation 


Each row of the mission platform location table defines the number of ISR 


platforms of a particular type assigned to a LRS (base). The MissionPlatformLocation 


elements are: 


characteristics 
e 


LRS: Unique identifier for the Launch and Recovery Site at which the 
MissionPackageTypes (ISR platform) are located. Each LRS can only be 
associated with one MissionPackageType (ISR Platform). Multiple LRSs 
can be associated with a single geographic location by using the 
UnitName element. 


UnitName: Unique name for a unit (base, airfield, or launch platform) 
which can be associated with multiple LRSs. 


PlatformType: Type of ISR platform at an LRS associated with a specific 
unit (base, airfield, or launch platform). 


Qty: Number of MissionPlatformTypes at the MissionPlatformLocation 
(unit). 


StartAvailable: The time that the mission platform becomes available, 
usually in H-hours. 


EndAvailable: The time that the mission platform is no longer available, 
usually in H-hours. 


Nn. MissionPlatformType 


The mission platform type table contains all the performance 
for the ISR platforms. The elements of the MissionPlatformType are: 
Name: The name of an individual type of ISR platform. 


TransitionTime: The amount of time, in hours, between the recovery of a 
platform and the time that it is available for a new mission. 


AirSpeed: The rate of travel of the ISR platform in meters per hour. 


ControlRadius: The maximum distance a platform requiring a Ground 
Control Station can be from a controlling CGS for effective control 
(Assumed to be via SATCOM and not used). 
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OperationalEnd: The maximum time a platform can be away from an 
LRS, the platform’s endurance. 


TimeHorizon: The time used for the optimization. Time horizon is 
usually 1 to 1.5 times the operating time. This value may need to be 
adjusted if transition time is greater than half the operating time. This 
value is critical to determining what missions will be considered in the 
look-ahead time window to launch a UAV if GCS capacity is available. 


MaxAltitude: Maximum operational altitude of the platform in meters. 
0. Mover 


The mover table enables the creation of different types of platforms. In 


this case only red or threat platforms (SAMs) are created. The mover platforms are 


assigned a name, a type (i.e., SAM), CVO type and an affiliation. Multiple copies of 


identical platforms can be created, if required. Note that ISR platform creation is done in 


a separate process in another table. The specific elements in the Mover table are: 


name: An identifier given to track agents within the simulation. If 
multiple platforms are created, they are given unique numbers in addition 
to the name. 


type: The fully qualified name of the implanting Mover class. This must 
be specified in the PLATFORM table of the database. 


qty: The number of identical platform that will be created. Defaults to 0. 


assignment: specifies whether the platform will be controlled by a fires 
CVO(C‘fires”), a sensor CVO(“sensor’’), both(“both’’) or none (“none”). 


affiliation: “Red” or “Blue” as listed in the SIDE table of the database 
(required). 


xLoc: The initial x coordinate of the agent unless a Box is used. If a Box 
is used, enter 0 as the initial x coordinate. 


yLoc: The initial y coordinate of the agent unless a Box is used. If a Box 
is used, enter 0 as the initial y coordinate. 


MaxSpeed: The maximum platform speed in meters per hour (not used). 


OperationalEndurance: The maximum operational endurance of the 
platform in hours (not used). 


formationLeader: Defines the lead platform when movers travel in 
formation (not used). 
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p. MoverManager 


The mover manager controls the movement of mover platforms when they 


are not given assignments from a CVO. This table must be completed for each mover 


even if none of the movers defined will actually travel. In this study, all of the mover 


platforms are fixed and do not move. The MoverManager attributes are: 


ID: An automatically generated number to track the different agents. 


class: The fully qualified name of the MoverManager class. The typical 
values are: 


e DAFS.platform.DAFSPathMoverManager 

e DAFS.platform.DAFSPatrolMoverManager 

° DAFS.platform.NAIMoverManager 

mover: The unique name given to the agent from the MOVE table. 


delay: Optional double value (must be >=0.0). Specifies the time after 0.0 
that the MoverManager will become active. If omitted, then an explicit 
call to start() 1s required to activate the MoverManager. 


startOnReset: Optional Boolean (‘“true” or “false”); if “true”, the 
specified MoverManager will commence its operation at the start of the 
simulation without any explicit command. If “false” or omitted, an 
explicit call to start() 1s required for the MoverManager to become active. 


q: Munition 


Munition is the definition for a weapon type consisting of a unique 


identification number, a noun name, and a mover assignment. The mover assignment 


determines the firing platform for the weapon. The operational parameters for the 


munition are characterized in the MunitionType table. The Munition elements are: 


ID: Automatically generated identification number for the individual 
munitions. 


Type: Name of the munition type. 


mover: Name of the mover (TEL, prime mover, aircraft, etc.) upon which 
the munition is loaded. 
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r. MunitionType 


The munition type table provides the parameters for an individual 
munition defined in the Munition table. Key attributes for the munition type are the 
maximum effective blast radius, the minimum and maximum effective ranges, and speed. 
The columns in the MunitionType table are: 

e MUNITION: A unique name for the ordinance. 

e WEIGHT: Weight in pounds of the weapon (not used). 
e MER: Effective blast radius in meters (minimum is 1.0). 
° MINRANGE: Minimum range in meters. 

e MAXRANGE: Maximum range in meters. 


e LOAD: The number of rounds included in a basic load out (initial 
quantity per platform). 


e SPEED: The speed of the round in meters per hour. 


° ALGORITHM: Determines what algorithm should be used dependent on 
type of munition. 


e BURST_SIZE: For indirect fires, defines the effects radius from impact 
(not used). 


e SUBMUNITION: Allows for submunitions and their effects (not used). 
S. PlatformType 


All non-ISR platform entities that are not mover entities in a scenario must 
be defined in the platform type table. SAM is the only PlatformType used in this study. 


The PlatformType elements are: 


° name: Link to mover. 

e value: Value used in fires CVO, set to zero and not used in this 
experiment. 
t. Route 


The route table provides a method for controlling the flight path of an ISR 
platform during travel between its home base (LRS/GCS) and the assigned mission area. 


Each entry in the table contains a single enumerated waypoint. Multiple entries are 
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required when more that one waypoint is necessary. If no routing is defined, the aircraft 


travel in a shortest path straight line between the base and the mission area. The Route 


elements are: 


StartLocation: Name of the LRS at which the mission originates. 
MissionArea: Name of the destination Mission Area for the route. 


PointNumber: Number indicting the sequence of the waypoint in the 
route. 


LocationX: The x coordinate of the waypoint. 


LocationY: The y coordinate of the waypoint. 
u. Sensor 


The sensor table creates a relationship between a mover and a sensor type 


for the sensors defined in the SensorType table. There is no relationship between the 


Sensor table and SensorRange table. The types of sensor relationships created in this 


study are strictly for red or threat platforms. The Sensor elements are: 


ID: AutoNumber used to track the sensors. 
type: As described under the SensorType Table. 


mover: The agent the sensor is assigned to. 
v. SensorRange 


Sensor range provides the maximum sensor range for the sensors to be 


carried by the ISR platforms. The types of sensors defined in this table consist of three 


elements, the platform type to carry the sensor, the sensor capability type, and the sensor 


range. This allows the creation of sensors of similar types with different ranges. For 


example, the EO sensor carried by a U-2 can a have a different range than an EO sensor 


mounted on a RQ-1. There is no relationship between the SensorRange table and Sensor 


or Sensor type tables. The sensors created in this table are strictly for blue platforms. 


Columns in the SensorRange table consist of: 


MissionPlatformType: Name of the platform that carries the sensor. 


MissionCapabilityType: Type of coverage provided by the sensor. The 
three capability types defined are EO/IR, SAR, and SIGINT. 


Range: Maximum range of the sensor in meters. 
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Ww. SensorType 


Sensor type creates a name and range for sensors that can be mounted on 
mover platforms. For this study, the SensorType table is used to create notional SAM 
fire control radars. There is no relationship between the SensorType table and 


SensorRange table. The SensorType elements are: 


e name: A unique name for the sensor. 
e maxRange: The maximum range of the sensor in meters. 
® class: The fully qualified class name for the sensor. 

x: Side 


Side determines the affiliation of the entities in the simulation (i.e., red or 
blue). The side table contains only one element: 


e name: Designates affiliation. Red indicates enemy and blue indicates 
own forces. 


y. SimEntity 


The SimEntity element is where platforms are defined. Each platform is 
defined in a Mover sub-element. 


® elements: The SimEntity has no attributes. 
3. Execution 


Simulation runs in JDAFS can be executed in a number of ways. Single runs, 
including multiple replications of the same scenario, can be made using either the JOAFS 
Graphical User Interface (GUI) (See Figure 6), or the command line. The 
implementation of more complex experiments requiring multiple replications of many 


design points should be run from a computing cluster. 
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# DAFS GUI (Simulation Initializing) 


Fie Simulation Mouse Mode Special Help 
urrent simulation time: 0.0000 Total entities being displayed: 212 


a il > Bp {x 
eal soe 1:23,083,144 





























-160 
Lat, Lon (9.588, -143,177) - x, y (32,1) 


Figure 6. | JOAFS Graphical User Interface 





4. Output 


Upon completion of each simulation, output tables are generated or populated 
depending on the method used to run the scenario. Single runs from the JOAFS GUI 
produce new Microsoft Access databases that must be manually saved following the 
completion of the simulation. Output from command line or computing cluster runs are 


saved back to output data tables in the original input database. 


JDAFS has the ability to create multiple output tables to capture MOE and model 
run parameters. These output tables include: Acquisition, Coverage, Coverage by Type, 
Coverage Delay, Killer-Victim Scoreboard, Mission Assignment, Run Information, and 
Schedule. Only the output tables capturing the MOEs of interest (Coverage, Coverage by 


Type, and Killer-Victim Scoreboard) for this research will be described in detail. For an 
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explanation of the other tables, see the ASC-U Users/Analyst Manual (Ahner et al. 


2006b) The 


following data tables and their associated elements are created at the 


completion of each run:&7 


a. Coverage 


The Coverage table contains one row for each Mission in the scenario. 


The elements in the Coverage table are: 


platform type. 


Replication: Number indicating the simulation run which provided the 
data. 


Mission: The name of the Mission. For this study, the mission name is a 
numerical code where the number before the decimal represents the 
mission name and the number after the decimal identifies the type of 
sensor required to cover the mission. 


Open Time: The total time, in hours, that the mission was vulnerable to 
coverage by the required sensor. 


Covered Time: The total amount of time, in hours that the mission was 
covered by the required sensor. 


Coverage: The percentage of time that the mission was covered by at least 
one sensor during the replication. 


Value Rate: The value received per unit time by the covering platform. 


Total Value: The total value obtained from this Mission during the 
scenario. 


b. Coverage by Type 


The Coverage by Type table contains one row for each Mission/ISR 
The following columns make up the Coverage by Type table: 


Replication: Number indicating the simulation run which provided the 
data. 


Mission: The name of the Mission. For this study, the mission name is a 
numerical code where the number before the decimal represents the 
mission name and the number after the decimal identifies the type of 
sensor required to cover the mission. 





6 Data table names, letter case and spelling are taken directly from the current version of the JDAFS 


input tables. 


7 Descriptions of the elements in the data tables are taken from the ASC-U Users/Analysts Manual. 
Where necessary, the wording has been updated to match the current usage in JDAFS. 
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Mission Platform Type: The type of platform assigned to the Mission. 


Open Time: The total time, in hours, that the mission was vulnerable to 
coverage by the required sensor. 


Covered Time: The total amount of time, in hours that the mission was 
covered by the required sensor. 


Coverage: The percentage of time that the mission was covered by at least 
one sensor during the replication. 


Value Rate: The value received per unit time by the covering platform. 


Total Value: The total value obtained from this Mission during the 
scenario. 


c. KillerVictimScoreboard 


The Killer-Victim Scoreboard provides a means of measuring attrition for 


a scenario. Each row in the table gives details on individual engagements. The Killer- 


Victim Scoreboard is composed of the following columns: 


Replication: Number indicating the simulation run that provided the data. 
Sim Time: Time in the simulation that the firing event occurred. 

Firing Platform: Name of the platform that launched the weapon. 
Munition: Type of munition fired by the platform. 


launch x: The x coordinate of the firing platform at the time of the firing 
event. 


launch y: The y coordinate of the firing platform at the time of the firing 
event. 


weight: Weight of the munition (not used). 

Target: The unique name of the platform targeted. 

target x: The x coordinate of the target at the time of the firing event. 
target y: The y coordinate of the target at the time of the firing event. 
type: Type of platform targeted. 

target value: Value of the target (not used). 

target side: Affiliation of the target platform. 


outcome: Result of the engagement, missed or killed. 
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Ul. SCENARIO DEVELOPMENT 


A. SCENARIOS 


The scenarios® built for this study are designed to conduct a trade-off analysis for 
Joint ISR operations. Two types of scenarios were generated for examination and 
comparison within JDAFS. The first scenario is a non-penetrating scenario (See Figures 
7 and 8, p. 35) where the ISR platforms do not penetrate the Country of Interest’s (COD 
national airspace. The internationally accepted buffer of 22 kilometers is respected on all 
flights and waypoints have been implemented to prevent ingressing and egressing aircraft 
from violating the COI’s sovereign airspace. The second scenario (See Figures 9 and 10, 
p. 36) assumes that conditions have changed to allow the violation of the COI’s airspace. 
With the incursions into the COI’s territory comes the risk of engagement by air defense 


assets, in this case surface-to-air missiles (SAMs). 


The country of interest for the scenario is depicted as a hexagon measuring 
1000km between any of its widest points. The size of the country allows the missions 
(ISR targets) to be widely dispersed and ensures that even the most capable sensors 
included in the study do not cover excessive portions of the country from a single mission 
area. The missions are dispersed non-uniformly throughout the country to achieve a 
sense of realism. While the mission dispersal is not uniform, it is also not random. Care 
is taken to place the missions such that the various mission sensor requirements are 
widely distributed to limit disproportionate coverage by a single platform. Mission areas 
were initially designed to ring the entire COI; however, allowing complete access to all 
countries surrounding a COI may be unrealistic. Overflight of mission areas along the 
western border and a portion of the southern border were deemed to be denied. Four 
operating bases for the ISR platforms were placed at varying distances in the non-denied 
areas surrounding the COI to allow for different transit times from base to mission area. 


The SAM sites were distributed across the COI with the strategic SAMs (SA-2 and SA- 


8 There is no intention to depict any real world operational area or threat order of battle. 


31 


10) treated as missions. The tactical SAMs (SA-6) were placed as point defense weapons 
at three separate mission types. See Figures 7, 8, 9, and 10 (pp. 35 and 36) for the 


specific placement of entities in this scenario. 


The design of a scenario for use in exercising the JOAFS model requires the 


creation of the following components: 


1. Missions (Targets) 


For the purposes of this simulation, 25 missions or targets of varying types were 
created and populated throughout the COI (See Figures 7, 8, 9 and 10, pages 35 and 36). 
Target types consist of command and control (C2) nodes, surface-to-air missiles (SAMs), 
short range ballistic missiles (SRBMs), medium range ballistic missiles (MRBM), long 
range ballistic missiles (LRBMs), airfields, weapons of mass destruction facilities (WMD 
FACs), military facilities (MIL FACs), and ammunition storage (AMMO STOR) 
facilities. Intelligence information for each type of target can only be collected against 
by specific types of sensors. The mission/sensor interactions must be explicitly created in 
the input tables and result in the creation of 55 target elements from the 25 actual 


missions (targets). The missions and their sensor requirements are listed in Table 1. 
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MISSION TYPE EO/IR | SAR | SIGINT 
1 (2 xX 
2 SA-2 Xx Xx xX 
3 WMD FAC Xx 
4 IRBM xX Xx xX 
5 AIRFIELD xX xX xX 
6 SRBM Xx Xx Xx 
7 C2 Xx 
8 C2 x 
9 SA-2 Xx Xx Xx 

10 SA-2 Xx xX Xx 
11 C2 Xx 
12 AMMO STOR Xx Xx 

13 AMMO STOR Xx Xx 

14 IRBM Xx xX xX 
15 LRBM Xx xX xX 
16 SRBM Xx xX xX 
17 MIL FAC Xx Xx 

18 WMD FAC Xx 

19 MIL FAC Xx xX 

20 MIL FAC Xx x 

21 IRBM x xX x 
22 MIL FAC Xx xX 

23 C2 Xx 
24 AIRFIELD Xx xX Xx 
25 SA-10 Xx xX xX 





Table 1. Targets and Required Sensors for Collection 


2: Mission Areas 


Mission areas (See Figures 7, 8, 9 and 10, pp. 35 and 36) are the locations created 
to simulate on-stations positions for the ISR platforms. The non-penetrating scenario has 
14 mission areas positioned outside the COI and the penetrating adds an additional 12 
mission areas inside the COI. The 14 non-penetrating mission areas are what remain of 
an originally plotted 24 mission areas after deleting the points deemed to be inaccessible 
due to being in a country that has denied overflight rights. For the placement of the 12 


additional penetrating mission areas, no regard was given to minimum risk flight routing 


in order to enable the examination of the effects of attrition. 
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3. Bases 


Four airfields or bases (See Figures 7, 8, 9 and 10, pp. 35 and 36) were created by 
assigning fixed common sets of coordinates to GCSs and LRSs. ISR platforms are then 
associated with specific GCSs and LRSs to establish the asset lay down at each base. 
The four bases are varying distances from the COI to take into account the differences in 


transit times that would be expected in real world operations. 


4. Threats 


The threats depicted in this scenario (See Figures 7, 8, 9 and 10, pp. 35 and 36) 
are all surface-to-air missiles (SAMs). Three different SAM systems of varying ranges 
were selected for inclusion in the scenario: the SA-2, SA-6, and SA-10. The P, for the 
SAMs was assumed to be the same against all ISR platforms. Studies conducted with 
real world operational data and parameters would have a separate P, for each individual 
ISR platform. The SA-6, a mobile platform, is used in fixed positions for this scenario. 
The SA-6s are not treated as missions or targets for this study, but are all collocated with 


missions. 


The operational parameters for the threat platforms used in JDAFS for this thesis 
are listed in Table 2. For real world studies conducted in a classified environment, 


operational parameters would be provided by the organization sponsoring the research. 














THREAT | MIN RANGE | MAX RANGE | MIN RANGE Pk | MAX RANGE Pk 
(meters) (meters) 
SA-2 1000 50000 0.6 0.5 
SA-6 1000 24000 0.6 0.5 
SA-10 1000 200000 0.6 0.5 























Table 2. Surface-to-Air Missile Parameters 


While not used in this scenario, other types of threat platforms can also be 


depicted in JDAFS such as, Anti-Aircraft Artillery (AAA) and fighter/interceptor aircraft. 
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A. TARGET (Mission) 
BASE 


O MISSION AREA 
22km AIRSPACE BUFFER, 


= COUNTRY BORDER 





Figure 7. | JOAFS Non-Penetrating Scenario as Designed (Best viewed in color) 





Figure 8. | JOAFS Non-Penetrating Scenario in JOAFS GUI (Best viewed in color) 
a3 


A TARGET (MISSION) 


O  ~MISSION AREA 
22km AIRSPACE BUFFER 


= COUNTRY BORDER 





Figure 9. | JDAFS Penetrating Scenario as Designed (Best viewed in color) 





Figure 10. JDAFS Penetrating Scenario in JOAFS GUI (Best viewed in color) 
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B. PLATFORMS 


The ISR platforms selected for this study are intended to represent a broad sample 
of theater and national level capabilities. It is important to note that the airframes used in 
this simulation are proxies for dissimilar ISR platforms to provide a sense of realism. 
The platforms are in no way, implicitly or explicitly, intended to reflect real world 


classified operational capabilities. 


1. RQ-1 Predator 




















Figure 11. RQ-1 Predator (From Global Aircraft 2007) 


The RQ-1 Predator is a medium altitude and endurance unmanned aerial vehicle 


that carries EO/IR and SAR sensors. 


De RQ-4 Global Hawk 











Figure 12. RQ-4 Global Hawk (From Tech-Writer.net 2007, accessed 07 May 2007) 


The RQ-4 Global Hawk is a high altitude long endurance unmanned aerial vehicle 


that carries EO/IR, SAR and ELINT sensor payload packages. 
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3. P-3C Orion 





Figure 13. P-3C Orion (From Aerospaceweb.org 2007, accessed 07 May 2007) 


The P-3C (AIP) Orion is a manned all-weather day-night long endurance 
maritime patrol aircraft that carries EO/IR, SAR and ELINT sensor payload packages. 


4. RC-135 Rivet Joint 





Figure 14. RC-135 Rivet Joint (From Aerospaceweb.org 2007, accessed 07 May 
2007) 
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The RC-135 Rivet Joint is a manned all-weather day-night Signals Intelligence 
(SIGNINT) collection platform. 


5. U-2 Dragon Lady 


C3 —_ Ni TH 





Figure 15. U-2 Dragon Lady (From Aerospaceweb.org 2007, accessed 07 May 2007) 


The U-2 is a manned high altitude reconnaissance aircraft with EO/IR, SAR, and 
ELINT packages. 


The operational parameters used in JDAFS for this thesis are listed in Table 3.? 
For real world studies conducted in a classified environment, operational parameters 


would be provided by the organization sponsoring the research. 




















PLATFORM MAX SPEED ENDURANCE | TRANSITION 

ALTITUDE | (meters/hour) (hours) TIME 

(meters) (hours) 
RQ-1 8000 148160 24 1.5 
RQ-4 18500 629860 36 LS 
P-3 9000 422256 12 1.5 
RC-135 10700 805629 11 LS 
U-2 21500 824140 10 1.5 























Table 3. ISR Platform Performance Parameters 


9 The operational parameters for the aircraft in this study are approximations based on a variety of 
open-source references. 
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C. SENSORS 


The sensors used in this study are implemented as cookie-cutter sensors with a 
defined footprint and probability of detection of 1.0 (i.e., if a target falls with in a sensors 
footprint, it will be detected). Other sensor capabilities may be available in future 


versions of JDAFS. 


1. Electro-Optical/Infrared (EO/IR) 


Electro-optical and infrared sensors are combined for the purposes of this research 
due to their similarity in detection ranges. EO sensors are day only sensors that produce 
images based on visible light, similar to standard photographs. IR sensors are day/night 
sensors that produce images based on thermal differences. IR sensors typically perform 
better at night when the surrounding environment tends to be cooler than the target of 


interest. 








EO IMAGE IR IMAGE 


Figure 16. EO and IR Images over Bosnia (After Global Defense Review 1997) 


2. SAR 


The Synthetic Aperture Radar (SAR) sensor is an all weather day/night sensor 


that uses reflected radar energy to create an image of an area or items of interest on the 
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ground. Due to the characteristics of SAR, this sensor’s actual performance is more 
closely approximated by the cookie-cutter sensor implementation than either the EO/IR 


or SIGINT sensors. 





Figure 17. SAR Image of the Pentagon (After Sandia National Laboratories 2006) 


3. SIGINT 


Signals Intelligence (SIGINT) sensors monitor the electromagnetic spectrum for 
signals of interest based on the ISR platform’s sensor payload abilities and mission 
tasking. The SIGINT sensors for this study do not differentiate between communications 
intelligence (COMINT) and electronic signals intelligence (ELINT) payloads nor 
whether the target is emitting, only whether the target can be collected against based on 


the footprint of the cookie-cutter sensor. 


4. Sensor Ranges 


The sensor ranges used in JDAFS for this thesis are listed in Table 4 and 
displayed graphically in Figure 18. For real world studies conducted in a classified 
environment, the sensor ranges and detection parameters would be provided by the 


organization sponsoring the research. 


41 









































PLATFORM | SENSOR | RANGE (meters) 
p-3 EO/IR 75000 
p-3 SAR 200000 

RC-135 SIGINT 300000 
RQ-1 EO/IR 50000 
RQ-1 SAR 200000 
RQ-4 EO/IR 75000 
RQ-4 SAR 200000 
RQ-4 SIGINT 300000 
U-2 EO/IR 100000 
U-2 SAR 250000 
U-2 SIGINT 400000 














Table 4. | ISR Platform Sensor Ranges 


U-2 RQ-4 


RC-135 P-3 RQ-1 
Figure 18. Cookie-Cutter Sensor Footprints 
D. CONSTRAINTS, LIMITATIONS, AND ASSUMPTIONS 


The use of modeling and simulation inherently involves constraints, limitations, 
and assumptions. As defined in the ASC-U User’s Manual, “Constraints are conditions 
imposed upon the development that are not under the control of the developers. 
Limitations are self-imposed and are due to choices that the developers of ASC-U made 
to limit the scope of the problem. Assumptions are those choices made to simplify the 


problem for ease of solution” (Ahner et al. 2006c, 9). 


1. Constraints 
e JDAFS improvements are limited to those achievable in a three month 
development window. 


e This study does not consider cueing from other assets that indicate a need 
for greater fidelity in the resolution on ISR information required. 


e No space or ground based assets are included in this study. 
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Limitations 


Performance data is not readily available due to a lack of 
performance/capability databases and classification of most relevant data 
beyond the UNCLASSIFIED level. 


No consideration is made of weather or environmental effects on sensor 
performance. 


LOS is based on visual horizon calculations only, no terrain features are 
modeled. 


The number and location of targets is fixed. 
Collection platforms operate only at fixed maximum platform altitudes. 


Collection platforms operate at fixed speeds, there is no difference 
between patrol and ingress/egress speeds. 


The model currently has a limited number of entities that can be included 
in the simulation without overwhelming the LpSolver routine. The 
maximum number of entities varies based on the type of scenario and the 
optimization interval. 


Platforms can not detect targets during ingress or egress. 


Platforms cannot travel from mission area to mission area. 
Assumptions 


The main factor in determining success of sensors is line-of-sight to target. 
The effects of weather and environmental factors are neglected. These do 
not significantly effect SAR sensors, but can significantly impact EO/IR 
sensors. 


CONOPS of airborne platforms is at maximum altitude perpendicular to 
mission area location. 


No camouflage, concealment, or deception is employed. 

At least two operating bases will be available. 

Platforms penetrating adversary national air space will be attacked. 
Platforms outside adversary national air space cannot be attacked. 
Platform speeds are constant. 

Cycle time between missions is constant. 


Platforms do not have emergent maintenance problems resulting in lost 
sorties. 


The number of targets is constant, there are no pop-up targets. 
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There are no mobile targets. 
All UAV control and datalink requirements are met. 


Atmospheric conditions do not affect collection, control, or transmission. 
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IV. MOE, DOE, AND RUN EXECUTION 


A. MEASURES OF EFFECTIVENESS (MOE) 


Three primary MOEs are of interest for analysis in this experiment: Coverage, 


Coverage by Type, and Attrition. The attrition MOE is only applicable to the penetrating 


scenario since, by design, the ISR platforms cannot be attacked if they do not violate the 


COIs airspace. JDAFS records numerous other data elements that could contribute to 


the analysis of other MOEs, as required. 


1. 


Coverage 


Coverage is the amount of time each Mission is covered by an ISR platform. In 


support of this MOE, JDAFS records the following (See Figure 19): 


Replication: Number indicating the simulation run which provided the 
data. 


Mission: The name of the Mission. For this study, the mission name is a 
numerical code where the number before the decimal represents the 
mission name (See Table 1) and the number after the decimal identifies 
the type of sensor required to cover the mission (x.1 = EO/IR, x.2 = SAR, 
x.3 = SIGINT). 


Open Time: The total time, in hours, that the mission was vulnerable to 
coverage by the required sensor. 


Covered Time: The total amount of time, in hours, that the mission was 
covered by the required sensor. 


Coverage: The percentage of time that the missions were covered by at 
least one sensor during the replication. 


Value Rate: The value received per unit time by the covering platform. 


Total Value: The total value obtained from this Mission during the 
scenario. 


[ replication uission open tinele coveredTime| coverage |valueRate|totalValue 


Figure 19. 


1 240, = 220.22436 0.917601 1, 220.2244 
1 21 240, = 67.872923) 0.282804 1, 67.87292 
1 2.2 240 68.5271731, 0.28553 1 68.52717 
1 2.3 240 227.174711 0.946561 1, 227.1747 
1 3.1 240 0 0 1 0 
1 4.1 240 220.782606 0.919928 1, 220.7826 


Coverage Data Table Example 
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2. 


Coverage by Type 


Coverage by Type is the amount of time each Mission is cover by each type of 


ISR platform. In support of this MOE, JDAFS records the following (See Figure 20): 


Replication: Number indicating the simulation run which provided the 
data. 


Mission: The name of the Mission. For this study, the mission name is a 
numerical code where the number before the decimal represents the 
mission name (See Table 1) and the number after the decimal identifies 
the type of sensor required to cover the mission (x.1 = EO/IR, x.2 = SAR, 
x.3 = SIGINT). 


Mission Platform Type: The type of platform assigned to the Mission. 


Open Time: The total time, in hours, that the mission was vulnerable to 
coverage by the required sensor. 


Covered Time: The total amount of time, in hours that the mission was 
covered by the required sensor. 


Coverage: The percentage of time that the mission was covered by at least 
one sensor during the replication. 


Value Rate: The value received per unit time by the covering platform. 


Total Value: The total value obtained from this Mission during the 
scenario. 


[replication] Mission] MissionPlatformType totalValue 
1 1.3 P-3 240 0 0 1 0 


Figure 20. 


3. 


—- ooo 


1.3 RC-135 240 19.21385271) 0.08005772 1 19.21385271 
1.3, RQ-1 240 0 0 1 0 
1.3, RQ-4 240 213.5448028) 0.889770012 1 213.5448028 
1.3)U-2 240, 19.0502298) 0.079375957 1. 19.0502298 
2.1 P-3 240 48.82269319) 0.203427888 1 48.82269319 


Coverage by Type Data Table Example 


Attrition 


Attrition represents the number and type of ISR platforms lost. In support of this 


MOE, JDAFS records the following in the KillerVictimScoreboard table (See Figure 21): 


Replication: Number indicating the simulation run that provided the data. 
Sim Time: Time in the simulation that the firing event occurred. 
Firing Platform: Name of the platform that launched the weapon. 
Munition: Type of munition fired by the platform. 
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e launch x: The x coordinate of the firing platform at the time of the firing 


event. 

e launch y: The y coordinate of the firing platform at the time of the firing 
event. 

e weight: Weight of the munition (not used). 

e Target: The unique name of the platform targeted. 

e target x: The x coordinate of the target at the time of the firing event. 

e target y: The y coordinate of the target at the time of the firing event. 


e type: Type of platform targeted. 



































° target value: Value of the target (not used). 
° target side: Affiliation of the target platform. 
e outcome: Result of the engagement, missed or killed. 
[Replication| SimTime [Firing Platform[Munition[launch_x[launch_y|weight] Target | target_x target_y _| type |target_value|target_side| outcome 
1) 0.826690884'SA-101 SA-10 0 1/U-2_1 0 168690.975 U-2 0 Blue killed 
1, 2.523543282|/SA-6_37 SA-6 50000 -350000 1 RQ-1_13) 72226.3106 -348622.3541 RQ-1 0 Blue killed 
1, 2.718760643|/SA-6_ 37 SA-6 50000 -350000 1U-2.9 70792.75866 -344680.0863 U-2 0 Blue missed 
1, 2.945224124)/SA-101 SA-10 0 1U-2.9 7010.649004 -169279.2848 U-2 0 Blue killed 
1, 4.142868882|SA-2_12 SA-2 -250000, 250000 1 RQ-1_1 | -261357.3946 294615.5364 RQ-1 0 Blue killed 
1, 8.206269191|SA-101 SA-10 0 0 1 RQ-1_14) 74908.41726 -168360.7734 RQ-1 0 Blue missed 


Figure 21. Killer Victim Scoreboard Example 
B. DESIGN OF EXPERIMENT 


A design of experiments is a structured scientific approach to determining how 
various input parameters, or factors, impact the results of an experiment. In order to fully 
explore a model’s output, a plan must be in place to ensure the collection of data 
representative of the model’s entire response surface. In computer simulation 
experiments, the understanding of model behavior and response is often best done 


through the use of data farming. 


The concept of data farming originated in 1998 and was further developed by 
Marine Corps Warfighting Laboratory’s Project Albert. Data farming is built on 
“distributed and high-performance computing; agent-based simulations and rapid model 
development; knowledge discovery methods; high dimensional data visualization 


techniques; design-of-experiments methods, human-computer interfaces; team work and 


47 


collaborative environments; and heuristic search techniques” (Horne and Meyer 2004, 
807-808). This collection of techniques and enabling technologies allows a 


comprehensive examination of the model. 


JDAFS is not currently configured to take full advantage of the data farming 
concept, but is likely to include future updates to integrate data farming features. For this 
study, the design of experiments, distributed computing, post-processing of output, and 


analysis all occur in a piecemeal manner. 


Ae Factors 


Given a database input format and instance (scenario), any possible data element 
in the database is a potential design factor. Practically speaking, however, for a model 
such as JDAFS, large groups of data elements together can constitute a single factor. The 
factors to be varied in this study are the number of each type of aircraft assigned to each 
base and the optimization interval. Of the 21 total factors, there are 12 integer factors 


with 4 levels, 8 integer factors with 7 levels, and | continuous factor. 


a. Assets Assigned to Bases 


The number of each type of aircraft assigned to each base make up the 
majority of factors for this experiment. There are 4 bases (A, B, C, and D) and 5 types of 
aircraft (RQ-1, RQ-4, P-3, RC-135, and U-2), for a total of 20 “Assets Assigned to Bases’ 
factors. This factor type is exclusively integer since is not possible to have fractional 
aircraft. A listing of all of the types of aircraft assigned to each base and the ranges for 


each factor can be found in Table 5. 
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Table 5. 


optimization events. 
Optimization intervals of greater than 24 hours do not make sense operationally, but there 
is nothing that prevents an analyst from choosing a longer interval. For optimization 
intervals that exceed the set scenario length, only an initial optimization would be 
determined and the simulation would run to completion based on this condition. The 
range of the optimization interval was limited to between 0.5 and 6 hours for this 


experiment (See Table 6). 



























































FACTOR MAX | MIN TYPE 
BASE A — RQ-4 3 0 INTEGER 
BASE A —- RQ-1 6 0 INTEGER 
BASE A - P-3 6 0 INTEGER 
BASE A — RC-135 3 0 INTEGER 
BASE A - U-2 ) 0 INTEGER 
BASE B — RQ-4 3 0 INTEGER 
BASE B - RQ-1 3 0 INTEGER 
BASE B — P-3 6 0 INTEGER 
BASE B — RC-135 6 0 INTEGER 
BASE B - U-2 3 0 INTEGER 
BASE C —- RQ-4 3 0 INTEGER 
BASE C - RQ-1 3 0 INTEGER 
BASE C - P-3 6 0 INTEGER 
BASE C — RC-135 6 0 INTEGER 
BASE C - U-2 3 0 INTEGER 
BASE D —- RQ-4 3 0 INTEGER 
BASE D-—-RQ-1 3 0 INTEGER 
BASE D - P-3 6 0 INTEGER 
BASE D — RC-135 6 0 INTEGER 
BASE D - U-2 3 0 INTEGER 

















Factors: Assets Assigned to Bases 


Optimization Interval 


intervals of less that 0.5 hours result in excessive runtimes. 


Table 6. 


Optimization interval is the elapsed time in the simulation between 


This factor can take on any continuous value greater than zero. 


Preliminary JDAFS test runs indicate that optimization 
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OPTIMIZATION INTERVAL 
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0.5 





CONTINUOUS 





Factor: Optimization Interval 


49 





2. NOLH 


Any number of DOE constructs could have been utilized to perform this 
experiment, such as, sequential bifurcation, folded sequential bifurcation, 2‘? fractional 
factorial, combined designs, or Nearly Orthogonal Latin Hypercube (NOLH) designs, 
among others (Sanchez 2006; Kleijnen et al. 2005). However, given the number of 
factors in this study, not all of the DOE applications would have been tractable. For 
example, a full factorial design with only 3 levels for each factor would result in 37! or 
10,460,353,202 design points. Assuming, only the deterministic non-penetrating scenario 
was to be executed and each run took a nominal 5 minutes to complete, the total 
processing time required would be nearly 1,000 years. Fortunately, alternatives exist that 
keep the run time at a practical level while ensuring a meaningful exploration of the 


response surface. 


The Nearly Orthogonal Latin Hypercube is such a design. The NOLH design 
provides a number of highly desirable DOE qualities including: orthogonality, space 
filling and bias protection, efficiency, and flexibility. 


e Orthogonality!°, or near orthogonality, is important to ensure 
independence of the regressors, or coefficient estimates in regression 
models. This property significantly enhances the ability to evaluate the 
suitability of factors for inclusion in the resultant metamodel and 
determine the overall contribution to metamodel fit (Kleijnen et al. 2005). 


e Space-filling designs enable the exploration of all regions of the response 
surface, not just the edges. By distributing the design points throughout 
the experimental region, analysts can minimize the number of assumptions 
made about the response surface. Additionally, “Space-filling designs also 
provide flexibility when estimating a large number of linear and nonlinear 
effects, as well as interaction, and so provide general bias protection when 
fitting metamodels of specific forms” (Kleijnen et al. 2005, 274). 


e The NOLH design provides efficiency in allowing the rapid determination 
of the correct minimum number of design points necessary for an 
experiment. Additionally, with the use of a NOLH design template, an 
experimenter can quickly produce a design containing numerous factors 
with multiple or continuous levels. 


10 Orthogonality exists when the inner product of any two columns of a matrix is equal to zero. 
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e The NOLH design is extremely flexible. Factors of interest can rapidly be 
added or levels can be changed to provide more insight into a model or to 
create entirely new designs. NOLH designs can also be stacked to achieve 
more desirable pairwise correlation or specific design points of interest 
can be appended to any existing design. 


For this study, a NOLH DOE was constructed using the NOLHDesigns_v4.xls 
worksheet (See Figure 22) created by Dr. Susan Sanchez (2005) based on algorithms 
developed by Cioppa (2002). The worksheets have been crafted specifically to avoid the 
problems associated with multicolinearity and ensure a robust space-filling design. This 
worksheet, capable of handling up to 29 factors, is available at < http://harvest.nps.edu/>. 
Five templates are included in the workbook to facilitate designs with various numbers of 
factors; 8-11, 12-16, 17-22, and 23-29. Each template can be used for any number of 
factors up to the maximum allowed in the template. This allows smaller experiments to 
be created with more than the minimum number of design points, improving upon the 
space-filling properties of the design and increasing the number of degrees of freedom for 
the output data. In this case, the 23-29 factor worksheet was used to create 257 design 


points!! based on the 21 factors in the experiment. 





11 The maximum number of factors in a Latin Hypercube is determined by m+combination(m —1, 2) 
where m is an integer greater than 1. Since the maximum number of factors for the chosen design template 
is 29, solving the previous equation for m, yields m = 8. The number of design points, n, can be found by 
n=2"+1. Therefore, a design that includes 29 factors requires 257 design points. 
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Figure 22. Nearly Orthogonal Latin Hypercube Worksheet. 


The design created consists of a 257 x 21 matrix with a maximum pairwise 
correlation of 0.0942.!213 Figure 23 provides a scatterplot of the design factors 
demonstrating the space-filling properties of the NOLH design. The uniform appearance 
of the points in the scatterplot is due to the integer structure of the 20 Assets Assigned to 





12 Ideally, the maximum pairwise correlation should be less than 0.03 to classify a matrix as nearly 
orthogonal. The 0.0942 achieved in the design for this study is low enough given the resolution of the 
model. Creating a design with a maximum pairwise correlation with the factors used in this study would 
have required more than doubling the number of design points. The runtime for this new design would 
have been impractical. (Cioppa 2002). 


13 Lower correlation numbers can be achieved, if required, by creating a stacked NOLH design. This 
is done by pasting the values from the NOLH worksheet into a new worksheet. Then, randomly permuting 
the column headings in the original NOLH worksheet and pasting those values into the new worksheet 
containing the values from the original worksheet, ensuring that the values in the columns are matched to 
the correct column headings. This results in a design with the same number of columns, but twice as many 
rows as the original design. 
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Bases factors. The space-filling patterns would be dramatically different if the factors 
were continuous in nature. An additional 17 special case design points were added, 


resulting in a 274 x 21 final design matrix. 


The special case design points are added to explore the response at specific 
combinations of factor settings that were not captured by the NOLH design. The first 15 
design points represent all possible combinations of 1, 2, 3, and 4 base configurations 
with maximum aircraft availability, 21 aircraft at each open base, and a midpoint setting 
of the optimization interval, 3.25 hours. The 2 remaining design points are for 4 base 
configurations with maximum aircraft availability and optimization intervals set at the 


minimum and maximum, 0.5 and 6 hours, respectively. 
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Scatterplot Matrix 
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Figure 23. Scatterplot Matrix for Design Factors 


Additional information on NOLH can be found in Cioppa’s (2002) Efficient 


Nearly Orthogonal and Space-Filling Experimental Design for High-Dimensional 


Complex Models doctoral dissertation. 
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C. EXECUTION OF SCENARIOS 
1. Implementing DOE 


In executing an experiment, the DOE determines the number of design points, but 
the number of replications must be decided upon by the analyst. A deterministic model 
needs only a single run of each design point. No matter how many times the experiment 
is run, a deterministic model always returns the same result. Selecting the number of 
replications necessary for a stochastic model is more challenging. Unfortunately, no 
clear rules exist that allow a determination of the “right” number of replications a priori. 
The number of replications must “gain enough data to achieve narrow confidence 
intervals and powerful hypothesis tests” (Sanchez 2006, 69). Conduct too few 
replications and the experiment may have to be rerun to get more data, too many 


replications and valuable resources may have been wasted (Chung 2004). 


For the purposes of this study, the non-penetrating scenario is purely deterministic 
and requires only one replication of each design point. The penetrating scenario, 
however, has the stochastic element of probability of kill (P,) for the SAMs. The use of 
10 replications of each design point for the penetrating scenario was all that was feasible 


given the scope of this research, processing constraints, and time limitations. 


A method was developed for JDAFS so that the analyst can specify which input 
factors are to be varied, match those factors with a design, and generate the 
corresponding input database. This was done so that there was no impact whatsoever on 
the basic input database design, and so that the specific experimental design used was 
completely interchangeable. That is, the analyst could keep the same basic input scenario 
and same set of factors, but swap in any experimental design scheme that was desired 
with only a minimal amount of changes. For each design point, the DesignGenerator 
(See Figure 24) program uses SQL statements to create a corresponding input file. 
JDAFS writes its output reports to the same database that was used for input, so there is 


always a direct association between input and output values. 
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Execution of JDAFS is oriented towards a single run with a single input file; there 
is no built-in capability in JDAFS itself to run experimental designs. Yet executing 
JDAFS with various input factors set based on efficient designs is essential to its effective 
use. The normal input to a JDAFS scenario is currently via an Access database; support 
for other databases such as MySQL, Oracle, and Derby will be added to future versions 


of the program. 


Setting a factor requires more than changing a single value; all data elements in 
columns corresponding to that factor must be changed to correctly set the factor to its 
appropriate level. In order to use the DesignGenerator, the values in the NOLH 
worksheet must be normalized to a range of [0, 1] and saved in plain-text comma- 
separated (csv) files. Conceptually, these files constitute tables in a virtual database (and 
future versions will be implemented as such); therefore, these tables of values are 
represented as a database called “Designs.” Different schemes can be used to generate 


these tables of values without modifying any other part of the input or the code. 


The Factors database consists of two tables: one to identify factors in the template 
database by table and column, together with a user-specified minimum and maximum 
value; and a second table that maps the number of factors (determined by the number of 
rows in the first table) to one of the design point files. That table is then read, and the 
corresponding values applied to the minimum and maximum values to produce an input 
value. The “DesignGenerator” program that does this is written in Java and uses Java 
Database Connectivity (JDBC) to access the databases. Hence, using different databases 


only requires that a different JDBC driver be installed. 


For each design point, the DesignGenerator program uses SQL statements to 
create a corresponding input file, which is numbered <name>xxxx.mdb, where “xxxx” is 
an index corresponding to the particular design point index. All the input from the 
template database is copied, and those entries that are factors are modified using the SQL 
UPDATE query to be set to the design point values. An additional table is written that 
identifies the particular names and values of the input factors. This information is used 
after JDAFS is executed to extract the independent and dependent variables for statistical 


analysis. 
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Template.mdb Factors.mdb Designs.mdb 


Table:Column # factors 


# Design Foints 
& valu 








Design 
Generator 











Input0000.mdb Input0128.mdb 


Figure 24. Design Generator for Producing Input Files 


2. Computing Resources 


While JDAFS is capable of running on standalone laptop or desktop computers, it 
is impractical to run simulation experiments requiring hundreds or possibly millions of 
runs, each potentially requiring several hours of runtime, on anything but a computing 
cluster. For this particular experiment, the non-penetrating scenario required 274 runs 
and the penetrating scenario required 2740 runs (274 x 10). These production runs were 
executed by Dr. Paul Sanchez, Senior Lecturer in the Operations Research Department at 
the Naval Postgraduate School, on the Simulation Experiments & Efficient Designs 
(SEED) Center Microsoft Windows computing cluster located at TRAC-Monterey on the 
NPS campus in Monterey, CA. The SEED Center computing cluster is composed of 12 
dual processor Dell desktops with 2G of RAM, running Windows XP. The machines are 
using Condor cluster software, which is open source software available from the 
University of Wisconsin <http://www.cs.wisc.edu/condor/>. The individual computers 
have Cygwin software <http:// www.cygwin.com/> installed to enable improved 
command line tools, remote access via Secure Shell, and sophisticated scripting. The 
non-penetrating scenario runs were completed in approximately 4 hours and _ the 


penetrating scenario runs in approximately 60 hours. 
af 


3. Post-processing of JDAFS Output 


Upon completion of the JDAFS model runs on the SEED Center computing 
cluster, the output data is written to tables in the input database. Therefore, 274 separate 
output databases are created, one for each design point in the scenario. By writing the 
output reports to the same database that was used for input, there is always a direct 
association between input and output values. The independent input/output databases 


must be combined into a consolidated database prior to performing any further analysis. 


The process for combining the independent output databases into a single 
common output database is essentially the reverse of creating the design databases. An 
OutputGenerator (See Figure 25) was created in JAVA using SQL statements to create a 
single output database. Like the DesignGenerator, JDBC is used to access the databases 
in the OutputGenerator. The OutputGenerator appends the information in the individual 
output database tables to like tables in the common output database. Since the JDAFS 
output data is written in string format, individual columns of data that should be numeric 


must be converted to the proper format prior to any data analysis. 


Output0000.mdb Output0274.mdb 





Output 
Generator 











Output.mdb 


Figure 25. Output Generator for Producing Output Files 
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The cumulative size of the output databases at the completion of the simulation 
runs was over 7.5 gigabytes. The process of combining the output databases into a single 
database for each scenario reduced the overall file size to approximately 470 megabytes. 
This was further reduced to just over 300 megabytes by converting the output database 


tables into tables in JMP. 
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Vv. ANALYSIS 


A detailed pre-simulation analysis of the scenarios and a post-simulation analysis 
of the output data collected from the execution of this study’s penetrating and non- 
penetrating scenarios were conducted. The following chapter describes the analysis 


process and relevant discoveries and points of interest from this research. 


A. ANALYSIS METHODOLOGY, TECHNIQUES, AND TOOLS 
ik Methodology 


The same analysis methodology is applied to both scenarios, non-penetrating and 
penetrating, in this experiment. Pre-simulation spreadsheet analysis was conducted to 
gain an understanding of how the scenario design would impact the JDAFS simulation 
results and to provide the capability to ensure reasonableness of JDOAFS output. Upon 
completion of the simulations, the data from each scenario was used to generate summary 
statistics to verify the validity of the output and identify outlying data points. The 
outliers were examined to determine their cause and potential impact on subsequent 
analysis. Multiple regression analysis was performed to allow the construction of 
metamodels for each scenario. The validity of these metamodels was confirmed through 
the use of regression and classification trees. Additional graphical analysis techniques 
were performed to gain insights and understanding into the behavior and influence of 


specific factors. 


2. Analytical Techniques 


A variety of analytical techniques were applied to fully examine and understand 
the scenarios and output from this study. The various techniques are complementary to 
one another and serve to validate the results of the methodology. The techniques applied 
in this study were spreadsheet, graphical, multiple regression, and classification and 


regression tree (CART) analysis. 
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a. Spreadsheet Analysis 


Spreadsheet analysis enables the rapid examination, manipulation, and 
display of large amounts of data in a flexible environment. The use of this tool facilitated 
the construction and evaluation of a variety of easily reconfigurable scenarios that were 
compatible with the JDAFS input databases and more sophisticated analysis tools. While 
not suitable for complex post-simulation data analysis, the use of spreadsheets enables a 


means to examine design scenarios prior to conducting simulation runs. 


b. Multiple Regression 


Regression is a technique that constructs a probabilistic model to examine 
the relationship between a dependent (response) variable and an independent variable 
(predictor) (Montgomery et al. 2004). Multiple regression involves more than one 
regressor or predictor and may contain interaction and quadratic terms. Models with first 
order predictors, interaction, and quadratic terms are referred to as full quadratic models. 
The response variable is represented as Y and the predictor variables as x,,x,,....x;. 
where k indicates the number of regressors. The f parameters, £,,/,,...,8,, are 
regression coefficients that determine the mean amount of change in Y for every one unit 
change in the associated predictor variable while all other predictors are held constant. 
The f, parameter is a constant term that indicates the point at which the regression line 
intersects the y-axis. The error term, €, is a “random variable that accounts for the 
failure of the model to fit the data exactly” (Montgomery et al. 2004, 1). Also known as 


the residual, ¢, is assumed to be normally distributed with a mean of 0 and a variance of 


o or N(0,o°). Devore (2004, 588) illustrates four common forms for multiple 
regression models containing two independent variables, x, and x,. 


1. First order model: 
Y=f.+ hx, +h.x, +8 
2. Second order no-interaction model: 


Y=f+ Bx, + Bx, + BG a Die +é 
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3: Model with first order predictors and interaction: 
Y=£,+6%+6,%,+ 6.4%, +6 
4, Full quadratic model: 


Y=$,+ 8x, + Bx, + BG + Ba + BX,X,+é 


Regression models must conform to several assumptions in order to be 
considered valid (Montgomery et al. 2004, 131): 
I. The relationship between the response y and the regressors is linear, at 
least approximately. 


The error term ¢ has a zero mean. 


2 

3 The error term € has a constant variance o°. 
4. The errors are uncorrelated. 
5 


The errors are normally distributed. 


These assumptions are summarized as follows: 


tid 


é,~N(0,0°), Vi. 


Conformity to these assumptions is demonstrated for each of the 
regression models constructed in this study. The linearity assumption between the 
response variable y and the regressors is checked by plotting the regression line against a 
scatter plot of the data, an actual by predicted plot. The assumptions of a zero mean, 
constant variance of o*, and uncorrelated errors are confirmed through a residual by 
predicted plot. This is indicated by a mean of zero and a cloud of apparently random data 
points. Finally, the normality assumption is demonstrated by histogram plot of the 


residuals overlaid with a normal curve. 
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For the purposes of this research, up to full quadratic models are 
constructed by stepwise regression. The wide variety of distances in mission to mission 
area pairings along with the differences in aircraft and sensor operational parameters 
result in inherent nonlinearities. These complexities must be accounted for through the 


use of full quadratic models. 


Cc. Classification and Regression Trees (CART) 


Classification and regression trees provide a method for recursively 
partitioning data sets in accordance with the relationship between the response variable 
and the predictors. Each split creates two branches in an inverted tree structure by 
considering all possible cuts or groupings and selecting the partition with the largest 
likelihood-ratio chi-square statistic (JMP Statistical Discovery Software 2006). This 
method generates a chart that is straightforward to interpret and clearly shows the most 


significant factors. 


d. Graphical Analysis 


Graphical analysis provides a visual representation of data allowing for 
rapid interpretation and enhanced understanding. Overlays, bar charts, scatter plots, line 
graphs, and contour plots were all use in the exploration of the pre- and post-simulation 
data. Additionally, the graphical representation of relevant findings is of benefit in 
conveying complex information to individuals or groups not intimately familiar with this 


topic or the research methods applied. 


3. Analysis Tools 
a. Microsoft Excel 


Microsoft Excel was used to perform pre-simulation spreadsheet analysis 
of the scenarios developed for this experiment in order to understand the type of output 
that might be generated by the actual JDAFS production runs. Additionally, Excel was 
utilized, where appropriate, for the production of explanatory tables and graphs in the 


analysis of the simulation output. 
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b. JMP Statistical Discovery Software 


The JMP (pronounce ‘jump’) Statistical Discovery Software from SAS is 
the tool chosen for analysis of the resultant data from the JDAFS model runs. This 
powerful analysis tool is intuitive to use and does not require additional programming or 
script writing to obtain results. JMP allows the analyst to interactively investigate data in 
a dynamically linked spreadsheet and graphical environment. Additionally, JMP 
provides a journaling function for saving and revisiting graphics or data tables of interest. 
More information on JMP Statistical Discovery Software can be found on the company’s 


web site: <http://www.jmp.com>. 


B. PRE-SIMULATION ANALYSIS 
iL. Non-Penetrating Scenario 


The non-penetrating scenario limits the collection ability of the ISR platforms to 
standoff mission areas. This restriction causes the implementation of waypoint routing to 
prevent COI overflight, thereby requiring additional flight time between bases and 


mission areas. 


a. Scenario Geometry 


As shown in Chapter III, Table 1, there are 25 targets, each with up to 3 
sensor requirements. These 25 targets are broken down into 20 EO/IR, 18 SAR, and 17 
SIGINT requirements, for a total of 55 missions. These 55 missions or sensor 
requirements can be serviced from any of the 14 mission areas in the non-penetrating 
scenario. (Satisfaction of sensor requirements depends upon the availability an ISR 
platform with the correct sensor package, sufficient sensor range and the platform’s 
ability to meet the minimum altitude to achieve line-of-sight.) By matching the 14 
mission areas with the 55 missions, 770 possible mission area-to-mission pairings are 
possible; 280 EO/IR, 252 SAR, and 238 SIGINT. See Figure 26 for an example of 
mission area-to-mission matching. Each of these pairings is taken into account by 


JDAFS in determining the optimum allocation of ISR platforms. 
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Figure 26. Example of Mission Area-to-Mission Pairings for a Single Mission Area 
to all EO/IR Missions 


It is expected that the lowest absolute distance and the lowest mean and 
median distance from a base to its associated mission areas will play a significant role in 
determining the overall coverage achieved by the various ISR platforms. An examination 


of the basing configuration as shown in Figure 27 and Table 7 reveals the following: 


e Base D is closest to its nearest mission area, then A, C, and B, 
respectively. 
e Base C has the lowest maximum distance to its farthest mission area, then 


D, A, and B, respectively. 


e Base D has the lowest mean and median travel distance to its mission 
areas, then C, A, and B, respectively. 


Note that waypoint routing affects the mean and median travel distance in the non- 


penetrating scenario since the COI airspace cannot be violated. 

















Figure 27. Basing Configuration 
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SUMMARY OF BASE TO MISSION 
AREA DISTANCE STATISTICS 
| A 


a = eae (a «ee > 


All distances are in meters 


Table 7. Non-Penetrating Scenario Summary of Base to Mission Area Distance 
Statistics (The additional transit required due to waypoint routing to prevent 
COI overflight is included in the distance calculations for this table). 


b. Sensor Coverage 


Each of the ISR platforms has a differing ability meet the various sensors 
demands. The requirement to collect from standoff mission areas in the non-penetrating 
scenario limits the amount of coverage each aircraft can provide. The following figures 


depict the potential coverage by each type of platform and sensor!4. 


Figure 28 illustrates the potential coverage by the RQ-1 EO/IR and SAR 
sensors. There are no EO/IR targets and ten SAR targets that can be ranged by the RQ-1. 


RQ-1 Sensor Coverage 





Figure 28. RQ-1 Sensor Coverage for the Non-Penetrating Scenario 


14 Note that the amount of coverage depicted in the illustrations may differ slightly from the amount of 
coverage that has been calculated due to the imprecise nature of the graphics. In all cases, the text refers to 
the calculated coverage. 
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Figure 29 demonstrates the ability of the RQ-4 to cover EO/IR, SAR and 
SIGINT sensor requirements. The RQ-4 can satisfy 2 EO/IR, 10 SAR, and 13 SIGINT 


requirements. 
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Figure 29. RQ-4 Sensor Coverage for the Non-Penetrating Scenario 


The ability of the P-3 to provide EO/IR and SAR coverage is illustrated in 
Figure 30. The P-3 can range 2 EO/IR missions and 10 SAR missions. 


P.3 Sensor Coverage 





Figure 30. P-3 Sensor Coverage for the Non-Penetrating Scenario 


Figure 31 depicts the coverage of the RC-135. The RC-135 has the ability 


to cover 13 SIGINT sensor requirements. 
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RC-135 Sensor Coverage 
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Figure 31. © RC-135 Sensor Coverage for the Non-Penetrating Scenario 


The U-2’s sensor capabilities are shown in Figure 32. The U-2 can satisfy 
7 EO/IR, 11 SAR, and 13 SIGINT missions, making it, sensor wise, the most capable 


platform in the scenario. 
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Figure 32. U-2 Sensor Coverage for the Non-Penetrating Scenario 


Table 8 provides a summary of the number of missions that can be 
covered by each ISR platform from each mission area. Ranked by ability to provide 
coverage of the most targets per mission area, the most capable aircraft is the U-2, 
followed by the RQ-4, the RC-135, the P-3, and finally the RQ-1. This further 
demonstrates the U-2’s superior ability to provide coverage. Note that the capability of 


the RQ-1 and the P-3 is very similar, with the P-3 marginally exceeding the RQ-1. 
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Table 8. | Non-Penetrating Scenario: Number of Missions Potentially Satisfied from 
Each Mission Area by Platform and Sensor Type 


Figure 33 compares the ability of the various ISR platforms to provide 
coverage. Note that the U-2 is superior to all other aircraft in every category. This 


difference is due to the U-2’s advantage in sensor range for all types of sensors. 
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Figure 33. Potential Mission Coverage by ISR Platform 
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Figure 34 illustrates the predicted time on station for each ISR platform 
from each base. The RQ-4 with its comparatively high speed and 36 hour mission 
duration is the clear leader in the ability to remain on-station. Interestingly, the RQ-1 
provides the least amount of time on-station per sortie. Despite a 24 hour mission 
duration, the RQ-4’s on-station endurance is limited by its slow speed. It is probable that 
the ability to cover a greater number of targets per sortie will be more important to 


overall coverage than the ability to remain on-station for a longer period of time. 
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Figure 34. Predicted Percentage of Time On-Station by Base and Platform Type 


The ability to satisfy all of the missions in the scenario is shown in Table 
9. There are 13 of 20 EO/IR, 7 of 18 SAR, and 4 of 17 SIGINT requirements that cannot 
be met by any platform. This indicates that only about 44% of the sensor requirements in 
the non-penetrating scenario can be satisfied. This does not mean that 44% is the 
maximum coverage number that can be achieved in the simulation. Since ISR platforms 
receive credit for all missions that are ranged regardless of whether or not they are 
assigned to that mission, the total coverage number will likely be higher than 44% due to 


duplication of mission coverage. 
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EO/IR Sensor Requirements 
Mission RQ-4 RQ-1 P-3 RC-135 U-2 Any 
1.1 *** NO SENSOR REQUIREME 


3 ** NO SENSOR REQUIREMENT *** 
.1 *** NO SENSOR REQUIREMENT *** 
0 Oo 1 1 


SAR Sensor Requirements 
Mission RQ-4 RQ-1 P-3 RC-135 U-2 Any 
1.1 *** NO SENSOR REQUIREME! 


2.1 

3.1 *** NO SENSOR REQUIREMENT ***| 

4.1 

5.1 

6.1 

7.1 *** NO SENSOR REQUIREMENT ***| 

8.1 *** NO SENSOR REQUIREMENT **“ 

9.1 3 3 3 0 3 «12 
10.1 1 1 0 2 5 
11.1 *** NO SENSOR REQUIREMENT *** 


1.3 2 0 0 2 3 7| 
2.3 1 0 1 2 4 
3.3 *** NO SENSOR REQUIREMENT ***| 


NwhHoanr+-f 
ooooo0oc°o°o 


11.3 


12.3 *** NO SENSOR REQUIREMENT *** 
0 13.3 *** NO SENSOR REQUIREMENT *** 
4 14.3 5 0 0 16 
0 15.3 1 0 0 
4 
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0 Oo 0 2 


16.3 4 0 O 14 


0 Oo 0 

3° 3 0 

0 O 0 

3.3 0 

. 3 id 0 17.3 *** NO SENSOR REQUIREMENT *** 
18.1 *** NO SENSOR REQUIREMEN 18.3 *** NO SENSOR REQUIREMENT *** 
3 19.3 *** NO SENSOR REQUIREMENT *** 
2. 20.3 *** NO SENSOR REQUIREMENT ***| 
1 21.3 2 0 O 2 4 8 
0 22.3 *** NO SENSOR REQUIREMENT *** 
R 23.3 0 

0 24.3 0 
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25.3 0 


0 
23.1 *** NO SENSO 
24.1 0 0 
25.1 0 0 
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Table 9. | Non-Penetrating Scenario: Ability to Satisfy Sensor Requirements by 


Platform Type (Cells highlighted in red indicate that the sensor requirement 
cannot be met by any platform type) 


Cc. Assessment 


The following assessments are made based on the preceding analysis of 


the factors affecting the non-penetrating scenario: 


° The U-2’s ability to cover the most missions per sortie coupled with its 
relatively high speed will make it the most significant asset in the 
simulation. 

° The RQ-4 will be important in maximizing coverage due to its long 


mission duration and relatively strong sensor package. 


° The relatively long range of the SIGINT sensor and the second best on- 
station performance may make the RC-135 a significant contributor to 
overall coverage. 


° The P-3’s relatively short sensor ranges and the lack of a SIGINT sensor 
will limit its contribution to mission coverage. 


° It is expected that the inability to cover any EO/IR targets will limit the 
utilization of the RQ-1 within the simulation. 


e The collective inability to range over 50% of the missions will severely 
limit the mean coverage. 
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Ze Penetrating Scenario 


The penetrating scenario allows the violation of the COI’s airspace, removing the 
standoff limitation and adding 12 additional mission areas. Waypoint routing is no 


longer uesed; however, the ISR platforms are now vulnerable to air defense threats. 


a. Scenario Geometry 


As shown in Chapter III, Table 1, there are 25 targets, each with up to 3 
sensor requirements. These 25 targets are broken down into 20 EO/IR, 18 SAR, and 17 
SIGINT requirements, for a total of 55 missions. These 55 missions or sensor 
requirements can be serviced from any of the 26 mission areas in the penetrating 
scenario. (Satisfaction of sensor requirements depends upon the availability an ISR 
platform with the correct sensor package, sufficient sensor range and the platform’s 
ability to meet the minimum altitude to achieve line-of-sight.) By matching the 26 
mission areas with the 55 missions, 1430 possible mission area-to-mission pairings are 
possible; 520 EO/IR, 468 SAR, and 442 SIGINT. See Figure 26, page 65 for an example 
of mission area-to-mission matching. Each of these pairings is taken into account by 


JDAFS in determining the optimum allocation of ISR platforms. 


Just as in the non-penetrating scenario, it is expected that the lowest 
absolute distance and the lowest mean and median distance from a base to its associated 
mission areas will play a significant role in determining the overall coverage achieved by 
the various ISR platforms. An examination of the basing configuration as shown in 


Figure 35 and Table 10 reveals the following: 


° Base D is closest to its nearest mission area, then A, C, and B, 
respectively. 
e Base D has the lowest maximum distance to its farthest mission area, then 


A, C, and B, respectively. 


e Base D has the lowest mean and median travel distance to its mission 
areas, then A, C, and B, respectively. 


Since no waypoint routing is necessary, the relative distances are constant from each base 
to its mission areas. That is, not only is Base D the closest to its nearest mission area, it 


also has the lowest mean distance to its mission areas. 
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Figure 35. Basing Configuration 





All distances are in meters 


Table 10. Penetrating Scenario Summary of Base to Mission Area Distance Statistics 


b. Sensor Coverage 


Each of the ISR platforms has a differing ability meet the various sensors 
demands. The requirement to collect from standoff mission areas in the non-penetrating 
scenario limits the amount of coverage each aircraft can provide. The following figures 


depict the potential coverage by each type of platform and sensor.!5 


Figure 36 illustrates the potential coverage by the RQ-1 EO/IR and SAR 
sensors. There are 3 EO/IR targets and 18 SAR targets that can be ranged by the RQ-1. 
Note that the RQ-1 can provide SAR coverage of the entire COI with the exception of a 


small area in the southwest corner. 


13 Note that the amount of coverage depicted in the illustrations may differ slightly from the amount of 
coverage that has been calculated due to the imprecise nature of the graphics. In all cases, the text refers to 
the calculated coverage. 
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RQ-1 Sensor Coverage 





Figure 36. RQ-1 Sensor Coverage for the Penetrating Scenario 


Figure 37 demonstrates the ability of the RQ-4 to cover EO/IR, SAR and 
SIGINT sensor requirements. The RQ-4 can satisfy 13 EO/IR, 18 SAR, and 17 SIGINT 
requirements. The RQ-1 can provide SAR coverage of the entire COI with the exception 


of a small area in the southwest corner and SIGINT coverage of the whole country. 


RQ-4 Sensor Coverage 


O@ 


SAR SIGINT 





Figure 37. RQ-4 Sensor Coverage for the Penetrating Scenario 


The ability of the P-3 to provide EO/IR and SAR coverage is illustrated in 
Figure 38. The P-3 can range 13 EO/IR missions and 18 SAR missions. Like the RQ-1 
and RQ-4, the P-3 can range all but a small portion of the country with its SAR sensor. 
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P.3 Sensor Coverage 





Figure 38. P-3 Sensor Coverage for the Penetrating Scenario 


Figure 39 depicts the coverage of the RC-135. The RC-135 has the ability 


to meet all 17 SIGINT sensor requirements and can cover the entire country. 
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Figure 39. RC-135 Sensor Coverage for the Penetrating Scenario 


The U-2’s sensor capabilities are shown in Figure 40. The U-2 can satisfy 
17 EO/IR, 18 SAR, and 17 SIGINT missions. The SAR and SIGINT sensor packages on 
the U-2 can range any location in the COI. Just as in the non-penetrating scenario, the U- 


2 is the most capable platform, sensor wise. 
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U-2 Sensor Coverage 
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Figure 40. U-2 Sensor Coverage for the Penetrating Scenario 


Table 11 provides a summary of the number of missions that can be 
covered by each ISR platform from each mission area. Ranked by ability to provide 
coverage of the most targets per mission area, the most capable aircraft is the U-2, 
followed by the RQ-4, the RC-135, the P-3, and finally the RQ-1. This further 
demonstrates the U-2’s superior ability to provide coverage. Note that the capability of 


the RQ-1 and the P-3 is very similar, with the P-3 marginally exceeding the RQ-1. 
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Table 11. Penetrating Scenario: Number of Missions Potentially Satisfied from Each 
Mission Area by Platform and Sensor Type 
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Figure 41 compares the ability of the various ISR platforms to provide 
coverage. Note that the U-2 is superior to all other aircraft in every category. This 


difference is due to the U-2’s advantage in sensor range for all types of sensors. 
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Figure 41. | Potential Mission Coverage by ISR Platform 


Figure 42 illustrates the predicted time on station for each ISR platform 
from each base. The RQ-4 with its comparatively high speed and 36 hour mission 
duration is the clear leader in the ability to remain on-station. Interestingly, the RQ-1 
provides the least amount of time on-station per sortie. Despite a 24 hour mission 
duration, the RQ-4’s on-station endurance is limited by its slow speed. The percentage of 
time on-station increases slightly for platforms at Bases A and D and decreases slightly 
for aircraft at Bases B and C when compared to the non-penetrating scenario due to the 
removal of waypoint routing. As in the non-penetrating scenario, it is anticipated that the 
ability to cover a greater number of targets per sortie will be more important to overall 


coverage than the ability to remain on-station for a longer period of time. 
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Predicted Percentage of Time On-Station by Base and Platform Type 
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Figure 42. Predicted Percentage of Time On-Station by Base and Platform Type 


The ability to satisfy all of the missions in the scenario is shown in Table 
12. There are 17 of 20 EO/IR, 18 of 18 SAR, and 17 of 17 SIGINT requirements that can 
be met. This indicates that approximately 95% of the sensor requirements in the scenario 
can be satisfied by the available platforms. This does not mean, however, that 95% is the 
amount of coverage that will be achieved in a simulation. The effects of transit time, 
attrition, and platform on-station time, among other factors will keep the total coverage 


from reaching 95%. 
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Penetrating Scenario: Ability to Satisfy Sensor Requirements by Platform 
Type (Cells highlighted in red indicate that the sensor requirement cannot be 
met by any platform type) 


Cc. Assessment 


The following assessments are made based on the preceding analysis of 


the factors affecting the penetrating scenario: 


The U-2’s ability to cover the most missions per sortie coupled with its 
relatively high speed will make it the most significant asset in the 
simulation. 


The RQ-4 will be important in maximizing coverage due to its long 
mission duration and relatively strong sensor package. 


The relatively long range of the SIGINT sensor and the second best on- 
station performance may make the RC-135 a significant contributor to 
overall coverage. 


The P-3’s relatively short sensor ranges and the lack of a SIGINT sensor 
will limit its contribution to mission coverage. 


It is expected that the minimal ability to cover EO/IR targets, along with 
the lack of a SIGINT sensor, will limit the utilization of the RQ-1 within 
the simulation. 


The fact that 95% of the sensor requirements can be met by the available 
platforms will dramatically improve coverage compared to the non- 
penetrating scenario. 
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C. MEASURES OF EFFECTIVENESS (MOE) 
The JDAFS simulation provides for the generation of a number of MOEs 
including, Coverage, Coverage by Type, and Attrition. 
1. Non-Penetrating Scenario Coverage 
a. Coverage Regression Model 


An initial plot of the distribution and summary statistics for 274 design 
points in the non-penetrating scenario is shown in Figure 43. The coverage results appear 
to be roughly normally distributed with a mean of 0.48. The five outlying data points at 


the bottom of the outlier box plot warrant further examination. 


Coverage 





a Quantiles Moments 
0.534) 


100.0% maximum 0.53257 Mean 0.4848123 
0.5147 —_ 99.5% 0.53218 Std Dev 0.0193636 
0.49 97.5% 0.52125 Std Err Mean 0.0011698 
90.0% 0.50897 upper 95% Mean 0.4871153 





0.47 75.0% quartile 0.49836 lower 95% Mean 0.4825093 
0.45 50.0% median 0.48424 N 274 
25.0% quartile 0.47352 
0.439 " 10.0% 0.46223 
0.4144 2.5% 0.44862 
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Figure 43. Distribution and Summary Statistics of Mission Area Coverage 


An examination of the data reveals that four of the outlying data points 1, 
2, 4, and 265 are the result of design points having only a single base available for 
operations (See Table 13). Even though each operating location had a full compliment of 
aircraft (21 total airframes), the mean coverage ranked at the bottom of all the results. 
When ranked from greatest to least coverage, D, C, A, then B, this ordering is not 
surprising when compared to the mean distance of the bases to their mission areas in the 
pre-simulation analysis. Having a single operating base available to ISR platforms in a 
region of the size encompassed in this study is unrealistic. Therefore, the design points 


that include only one operating location will not be considered for the remainder of this 
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analysis. The remaining outlier, data point 70, had ISR platforms at all locations (30 total 
airframes); however, having only one U-2 and a near maximum optimization interval 
appear to be the reason for the low mean coverage. There is no easy or obvious 


justification for removing this design point from the data set for further analysis. 
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Table 13. Non-Penetrating Scenario: Low Outlying Data Points 


For comparison with future models, a full quadratic model with the 
outlying data points was constructed. Figure 44 shows this model. Note that the R- 
squared! value for this model is 0.73. 
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Figure 44. Actual by Predicted Coverage Plot for Full Quadratic Model with Single 
Base Outliers Included. 


After removing the single base outliers, the distribution and summary 
statistics for the non-penetrating scenario data were recalculated. The increase in the 
mean coverage is negligible, but the remaining data more closely approximates a normal 


distribution (See Figure 45). 


16 R* or R-squared, the coefficient of determination, is defined as the proportion of response 
variation that is explained by the regression model (Devore 2004, 514). 
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Coverage 





Quantiles Moments 


100.0% maximum 0.53257 Mean 0.485819 
0.54 99.5% 0.53220 Std Dev 0.0175688 
97.5% 0.52137 Std Err Mean 0.0010692 
0.49 90.0% 0.50899 upper95% Mean 0.487924 
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0.5% 0.43413 
0.0% minimum 0.42671 
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Figure 45. Non-Penetrating Scenario: Distribution and Summary Statistics of Mission 
Area Coverage (single base outliers removed) 

From the final 270 design point dataset a full quadratic model with main 

effect, interaction, and polynomial terms was created. The Stepwise Regression Control 


settings within JMP were as follows: 


e Probability to enter: 0.01 
e Probability to leave: 0.01 
e Direction: Mixed 


The Construct Model Effects macros for Factorial to Degree and Polynomial to Degree 


were set at 2 to allow two-way interaction and quadratic terms. 


The resultant model Achieves an R-squared of 0.78 and contains 12 main 
effect terms, 4 interaction terms, and 3 second order terms. See Figures 46 and 47 for the 


regression plot and a list of the regression model terms. 
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Figure 46. Actual by Predicted Coverage Plot for Full Quadratic Model with Single 
Base Outliers Removed 
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Parameter Estimates 
Term Estimate StdError tRatio Prob>|t| 
Intercept 0.4762323 0.002843 167.51 <.0001* 
A RQ-4 -0.002767 0.000534 -5.18 <.0001* 
A RQ-1 -0.000885 0.000285 -3.10 0.0021* 
A RC-135 0.0039884 0.000535 7.45 <.0001* 
A U-2 0.0056309 0.000536 10.50 <.0001* 
B RQ-4 -0.002767 0.000541 -5.12 <.0001* 
B RC-135 -0.000835 0.000538 -1.55 0.1219 
BU-2 0.0035851 0.000534 6.71 <.0001* 
C RQ-4 -0.002511 0.000531 -4.73 <.0001* 
C U-2 0.0058404 0.000531 10.99 <.0001* 
D RQ-4 -0.001957 0.000529 -3.70 0.0003* 
D U-2 0.0074843 0.000532 14.07 <.0001* 
OPT INT -0.004388 0.000361 -12.15 <.0001* 
(A RQ-4-1.52963)*(B RC-135-1.52963) -0.002434 0.000522 -4.66 <.0001* 
(A RC-135-1.52963)*(D RQ-4-1.52963) -0.001769 0.000518 -3.41  0.0007* 
(B RC-135-1.52963)*(D RQ-4-1.52963) -0.001793 0.000535 -3.35 0.0009* 
(C RQ-4-1.52963)*(D U-2-1.52963) -0.002549 0.000504 -5.05 <.0001* 
(B RQ-4-1.52963)*(B RQ-4-1.52963) 0.001986 0.000574 3.46 0.0006* 
(C U-2-1.52963)*(C U-2-1.52963) -0.001624 0.00057 -2.85 0.0047* 
(D RQ-4-1.52963)*(D RQ-4-1.52963) 0.0030399 0.000559 5.44 <.0001* 


Figure 47. Full Quadratic Coverage Model Terms 


As expected, the most capable platforms, the U-2 and RQ-4, show up as 
terms from each base. The RC-135 is reflected twice from Bases A and B. The RQ-1 
from Base A is the only entry for that platform. Interestingly, the P-3 is not represented 
in the model at all. In addition to the aircraft factors, the optimization interval term is 


also included in the final model. 


To test the validity of the Coverage Full Quadratic Regression model, the 
regression assumptions from Part A of this chapter must be verified. The Actual by 
Predicted Plot in Figure 46 indicates that the relationship between the response variable 
(coverage) and the regressors is reasonably linear. The apparently random cloud of data 
points with a mean of zero in the Residual by Predicted Plot, Figure 48, confirms that the 
requirements for a zero mean, constant variance, and lack of correlation of the residuals is 
met. Compliance with the normality assumption is shown in Figure 49 by a histogram of 


the residuals overlaid with a normal curve and a normal quantile plot. 
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Figure 48. Residual by Predicted Plot for Coverage Model 
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Figure 49. Histogram and Normal Quantile Plot of Residuals for Coverage Model 


b. Coverage Factor Interactions 


To further examine the interactions of the factors in the model, an 
Interaction Profile Plot, Figure 50, was constructed. Each of the individual cells within 
the larger plot display the interaction between two factors and their effect on Coverage. 
Each cell contains the plot of two lines, one for the high setting and one for the low 
setting of the term found in the name box for that row. Solid lines or curves indicate the 
presence of interaction between terms. The lighter, broken lines or curves indicate no 
interaction. Note that the lines without interaction are nearly parallel. The y axis of the 
grid is the response variable, coverage, and the x axis indicates the levels for each factor. 
When assessing the change to the response variable based on moving along one of the 


lines in a single cell, it is assumed that all other factors are held constant. 
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Interaction Profiles 
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Figure 50. Interaction Profile for Final Coverage Model 


The following figures displaying terms with significant interaction are 
taken from the larger Interaction Profile plot, Figure 50. Note that in all cases, the 
coverage varies only marginally regardless of the settings of the interaction terms for this 
model. 


° Figure 51 shows that with no RQ-4s at Base A, total coverage actually 
increases with the addition of RC-135s at Base B. However, if there are 3 
RQ-4s at Base A, coverage declines as RC-135s are added at Base B. 
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Figure 51. 


Figure 52. 


Figure 53. 
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Interaction Profile for A RQ-4 and B RC-135 


Figure 52 shows that the RC-135s at Base A have a quadratic interaction 
with the addition of RQ-4s at Base D. That is, regardless of the number of 
RC-135s at Base A, coverage initially declines and then rises with the 
addition of RQ-4s at Base D. 
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Interaction Profile for A RC-135 and D RQ-4 


Figure 53 shows that with no RC-135s at Base B, total coverage actually 
increases slightly with the addition of RQ-4s at Base A. However, if there 
are 3 RC-135s at Base B, coverage declines as RQ-4s are added at Base A. 
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Interaction Profile for B RC-135 and A RQ-4 


Figure 54 shows that the RC-135s at Base B have a quadratic interaction 
with the addition of RQ-4s at Base D. That is, regardless of the number of 
RC-135s at Base B, coverage initially declines and then rises with the 
addition of RQ-4s at Base D. 
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Figure 54. Interaction Profile for B RC-135 and D RQ-4 


° Figure 55 shows that with the number of RQ-4s at Base C at zero or three, 
total coverage increases with the addition of U-2s at Base D. The slope of 
the line for zero RQ-4s is steeper, indicating that it is better to have no 
RQ-4’s at Base C than to have the maximum number. 
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Figure 55. Interaction Profile for C RQ-4 and D U-2 


° Figure 56 shows that with the number of RQ-4s at Base D at zero or three, 
total coverage increases with the addition of RC-135s at Base A. The 
slope of the line for zero RQ-4s is steeper, indicating that it is better to 
have no RQ-4’s at Base D than to have the maximum number. 
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Figure 56. Interaction Profile for D RQ-4 and A RC-135 
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e Figure 57 shows that with no RQ-4s at Base D, total coverage actually 
increases with the addition of RC-135s at base B. However, if there are 3 
RQ-4s at Base D, coverage declines as RC-135s are added at Base B. 
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Figure 57. Interaction Profile for D RQ-4 and B RC-135 


° Figure 58 shows that with no U-2s at Base D, total coverage increases 
slightly with the addition of RQ-4s at Base C. However, if there are 3 U- 
2s at Base D, coverage declines as RQ-4s are added at Base C. 
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Figure 58. Interaction Profile for D U-2 and C RQ-4 


In several of the cases illustrated above, coverage actually declines with 
addition of more aircraft. This is counter intuitive. Normally, the expectation is that 
more assets provide more coverage. The seeming anomaly is explained by the 
optimization that takes place within JOAFS. The optimization algorithm finds the best 
coverage possible based on matching the assets available to the missions to be serviced. 
In the simulation, only one platform can be assigned to a mission. Once that assignment 
is made no additional aircraft can be assigned to the same missions. The addition of 
more aircraft can result in inferior platforms being assigned to missions, thereby 


preventing more capable aircraft from having the opportunity for maximum assignment. 
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c. Coverage Regression Tree Analysis 


Another means of exploring the data and confirming the validity of the 
regression model is through the use of classification and regression trees (CART). To 
determine the number of splits that would continue to yield additional explanatory value 
for Coverage, 20 splits were done in rapid succession to record the R-squared value 
without regard to the content or structure of the regression tree. These R-squared values 
were plotted to find the point that the curve began to plateau, indicating diminishing 
returns from each successive split. In this case, 11 splits are warranted. Figure 59 


provides a plot of the R-squared values against the number of splits. 


Non-Penetrating CART Analysis 
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Figure 59. R-Squared vs. Number of Regression Tree Splits 


Figure 60 shows the regression tree for Coverage with 11 splits. The full 
tree and partition graph can be found in Appendix I. The first split indicates that better 
coverage is obtained when the optimization interval is 4.64 hours or less. The second and 
third splits are on the presence of U-2s. In the second split, coverage is improved by 


having two or more U-2s at Base D and in the third split coverage is improved by having 
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at least one U-2 at Base C. The fourth split indicates that better coverage is gained by 
having at least one RQ-4 at Base B. The fifth split continues to demonstrate the 
importance of the U-2, showing that it is beneficial to have two or more U-2s at Base A. 
The sixth split indicates that coverage is improved by the stationing at least one RC-135 
at Base A. The seventh split indicates that no RQ-4 platforms at Base A improves 
coverage. The eighth split is a return to the optimization interval, showing that coverage 
improves with optimization intervals of 3.03 hours or less. The most significant factor at 
the ninth split is U-2s at Base A, indicting that coverage is improved by having at least 
one U-2 at Base A. Split ten indicates improved coverage by stationing one or more RC- 
135s at Base A. The final split again is on U-2s at Base A, this time indicating improved 


coverage with one or more U-2s at Base A. 


The splits in the regression tree, especially the multiple splits on 
optimization interval, Base A U-2s, and the split on Base D U-2s, serve to reinforce the 


findings from the regression model. 







































































































































































































































































































































































Partition for Coverage 
RSquare NN Splits 
Included 0.529 270 11 
Excluded 0.099 4 
All Rows 
OPT INT>=4.64333333 OPT INT<4.64333333 
A U-2<1}}A U-2>=1 D U-2<2 D U-2>=2 
C U-2<1 C U-2>=1 B RQ-4>=1 B RQ-4<1 
A U-2<2 A U-2>=2| C RQ-4>=1 C RQ-4<1 
A RC-135<1}/A RC-135>=1 A RQ-4>=1 A RQ-4<1 
D RQ-4>=1 D RQ-4<1 
OPT INT>=3.03]| OPT INT<3.03 


























Figure 60. Regression Tree for Coverage 
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d. Coverage Contour Plot Analysis 


To further explore the significance of the U-2 as a primary driver of the 
level of Coverage in the non-penetrating regression model, a contour plot was generated 
to illustrate the how the number of U-2s relates to the total number of aircraft. The 
various color filled regions of the plot represent the amount of coverage attained as the 
number of U-2s and total aircraft vary. The combinatorial nature of the problem and 
strong interactions of the regressors, coupled with the fact that the other factors are not 
held fixed, results in a non-smooth contour plot. Additionally, the combinatorial nature 
and base locations along with the approximate dynamic programming technique cause 
“islands” in the contour plot. The contour plot is read by selecting a value on the x-axis, 
TOTAL ACFT, and then examining the corresponding coverage based on number of U-2 
platforms on the y-axis (i.e., given 30 total aircraft, as the number of U-2 platforms 
ranges between 1 and 9, coverage ranges from < 48.3% to > 50%). Figure 61 
demonstrates that in nearly all cases, coverage is improved by increasing the number of 
U-2 platforms. It appears that having three or less U-2’s dramatically limits the level of 
coverage while six or more tends to drive coverage up. The value of additional U-2 


assets peaks at 11. 


Contour Plot for Coverage 
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Figure 61. Coverage Contour Plot of Coverage for U-2 vs. Total Aircraft 
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Zz; Non-Penetrating Scenario Coverage by Type 


The coverage by type MOE is a subset of total coverage. Recall that mean 
coverage for the non-penetrating scenario was 48.6%. Of this total amount of coverage, 
the RQ-4 is the largest contributor at 38.2% of the total, slightly ahead of the U-2 at 
36.9%. These two platforms alone account for over 75% of the total coverage. This is 
not unexpected given the superior sensor packages carried by these airframes. It 
appeared in pre-simulation analysis that the RQ-1 would not be a primary contributor to 
total coverage given its relatively limited sensor range. This is clearly not the case. The 
long dwell time apparently gives the RQ-1 a significant advantage over the P-3 and the 
RC-135. The RC-135s total contribution is likely limited since it carries only the 
SIGINT sensor. This disadvantage is compounded by the fact that there are fewer 
SIGINT targets in the scenario than EO/IR or SAR targets. See Figure 62 for a 


comparison of coverage by platform type. 
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Figure 62. Coverage by Platform Type 
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3. Penetrating Scenario Coverage 
a. Coverage Regression Model 


An initial plot of the distribution and summary statistics for 274 design 
points in the penetrating scenario is shown in Figure 63. The coverage results appear to 
be roughly normally distributed with a mean of 0.48. The seven low outlying and four 


high outlying data points in the outlier box plot warrant examination. 








Coverage 
Quantiles Moments 
100.0% maximum 0.78934 Mean 0.588048 
99.5% 0.77638 Std Dev 0.0686427 
97.5% 0.71055 Std Err Mean 0.0041469 
90.0% 0.66028 upper95% Mean 0.5962119 
75.0% quartile 0.62264 lower95% Mean 0.5798841 
50.0% median 0.59131 N 274 
25.0% quartile 0.55980 
10.0% 0.51609 
2.5% 0.46004 
0.5% 0.22126 
0.0% minimum 0.22036 




















Figure 63. Penetrating Scenario: Distribution and Summary Statistics of Mission 
Area Coverage 


An examination of the data reveals that four of the low outlying data 
points 1, 2, 4, and 265 are the result of design points having only a single base available 
for operations (See Table 14). Even though each operating location had a full 
compliment of aircraft (21 total airframes), the mean coverage ranked at the bottom of all 
the results. Having a single operating base available to ISR platforms in a region of the 
size encompassed in this study is unrealistic. Therefore, the design points that include 
only one operating location are not considered in the remained of this analysis. The 
remaining outliers, data points 11, 38, and 27, had ISR platforms at all locations (17, 33, 
and 30 total airframes, respectively, out of a possible 84). The low total numbers of 
aircraft, coupled with low numbers of U-2 (3, 6, and 3, respectively) and RQ-4 (2, 2, and 
3, respectively) airframes appears to be the reason for the low mean coverage. There is 


no justification for removing these design points from the data set in further analysis. 
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Design P S A 7 A 5 5 5 5 5 » ) UV) ) » OP 
[Pont | no-s{ noes [r.)ne-a5|t2] nono] | nts] Ua) no] nos [Pa nc-95|u-2| naa] no]. ne-s05 2 | coverage 


pt Of ff fff ff ff ff 3) 3.25) 0.222771] 
ee aE 0. 220588 


a) EE 0.318629 


34 0.98] 0.404777 
28 3. 3 0. ae 


a RECOIL 
test 3} aot taf tot ott oot oot tot a8 0. 263008 








Table 14. Penetrating Scenario: Low Outlying Data Points 


Four high outlier data points were also identified, 86, 261, 273 and 274 
(See Table 15). Each of these points contains high numbers of U-2 and RQ-4 aircraft, in 
fact, high numbers of all aircraft (56, 68, 84, and 84 total aircraft, respectively out of a 
possible 84). These design points, while possibly unrealistic in terms of available ISR 
platforms, do not violate any of the assumptions of the modeling scenario and do not 


warrant removal from the data set. 


Vesign A - A p A 5 5 5 5 5 J) ) D J » OP 
[Point |no-s{nos/p.9|no-a seers Rov eee Rov seers cso 2 | coverage 





86 5.35| 0.728791 
261 25. = 0.754785 


i aa SE 0.749603 
tart toot tot tt ttt ot ot tot tt 0.789344 





Table 15. Penetrating Scenario: High Outlying Data Points 


After removing the single base outliers, the distribution and summary 
statistics for the penetrating scenario data were recalculated (See Figure 64). The mean 
coverage increases by approximately 1% and the remaining data more closely 


approximates a normal distribution. 


Coverage 





Quantiles Moments 


100.0% maximum 0.78934 Mean 0.5929633 
99.5% 0.77708 Std Dev 0.0556522 
97.5% 0.71091 Std Err Mean 0.0033869 
90.0% 0.66040 upper95% Mean 0.5996315 
75.0% quartile 0.62300 lower95% Mean 0.5862951 
50.0% median 0.59176 N 270 
25.0% quartile 0.56045 
10.0% 0.52025 
2.5% 0.48639 
0.5% 0.41733 
0.0% minimum 0.40478 























Figure 64. Penetrating Scenario: Distribution and Summary Statistics of Mission 
Area Coverage (single base outliers removed) 
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From the final 270 design point dataset a full quadratic model with main 
effect, interaction, and polynomial terms was created. The Stepwise Regression Control 


settings within JMP were as follows: 


e Probability to enter: 0.01 
e Probability to leave: 0.01 
e Direction: Forward 


The Construct Model Effects macros for Factorial to Degree and Polynomial to Degree 


were set at 2 to allow the possibility of two-way interaction and quadratic terms. 


The resultant model achieves an R-squared of 0.71 and contains 16 main 
effect terms, 7 interaction terms, and 4 second order terms. See Figures 65 and 66 for the 
regression plot and a list of the regression model terms. Note that the optimization 


interval (OPT INT) is clearly the most significant factor in the model. 
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Figure 65. Actual by Predicted Coverage Plot for Full Quadratic Model 
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Parameter Estimates 
Term Estimate StdError tRatio Prob>|t| 
Intercept 0.3606258 0.011655 30.94 <.0001* 
A RQ-4 0.0054156 0.001988 2.72 0.0069* 
A RQ-1 0.0065338 0.001058 6.17 <.0001* 
A P-3 0.0061525 0.001063 5.79 <.0001* 
A U-2 0.0059377 0.001998 2.97 0.0033* 
B RQ-4 0.0080442 0.00197 4.08 <.0001* 
B P-3 0.003962 0.001046 3.79 0.0002* 
BU-2 0.0064787 0.001974 3.28 0.0012* 
C RQ-4 0.0107559 0.001982 5.43 <.0001* 
C P-3 0.0022811 0.001052 2.17 0.0311* 
C U-2 0.0050889 0.001977 2.57 0.0107* 
D RQ-4 0.0063526 0.001993 3.19 0.0016* 
D RQ-1 0.003488 0.001069 3.26 0.0013* 
D P-3 0.0081541 0.001063 7.67  <.0001* 
D RC-135 0.0071914 0.002009 3.58 0.0004* 
D U-2 0.0136722 0.002008 6.81 <.0001* 
OPT INT 0.0139716 0.001338 10.45 <.0001* 
(A RQ-4-1.52963)*(OPT INT-3.46986) -0.003346 0.001438 -2.33 0.0208* 
(A RQ-1-3.05926)*(D U-2-1.52963) 0.0032962 0.001103 2.99 0.0031* 
(A P-3-3.05926)*(C RQ-4-1.52963) 0.0009105 0.000949 0.96 0.3383 
(A U-2-1.52963)*(D RQ-4-1.52963) -0.004324 0.001949 -2.22 0.0274* 
(B RQ-4-1.52963)*(B U-2-1.52963) -0.003686 0.002221 -1.66 0.0983 
(C RQ-4-1.52963)*(OPT INT-3.46986) -0.003959 0.001359 -2.91 0.0039* 
(C P-3-3.05926)*(C U-2-1.52963) -0.001152 0.001217 -0.95 0.3448 
(D RQ-1-3.05926)*(D RC-135-1.52963) -0.003878 0.001108 -3.50 0.0005* 
(D P-3-3.05926)*(OPT INT-3.46986) 0.0006726 0.000747 0.90 0.3688 
(B P-3-3.05926)*(B P-3-3.05926) -0.001511 0.000673 -2.25 0.0256* 
(D U-2-1.52963)*(D U-2-1.52963) -0.008468 0.00225 -3.76 0.0002* 





Figure 66. Full Quadratic Coverage Model Terms 


The full quadratic model for the penetrating scenario differs significantly 
from the non-penetrating scenario model in the number of terms included. In the 
penetrating model all of the platform types show up at least once, where as in the non- 
penetrating model, P-3 were not represented at all. The main similarity between the non- 
penetrating model and the penetrating model is that the U-2 and RQ-4 are represented at 


every base as significant dependent variables in the regression equation. 


To test the validity of the Coverage Full Quadratic Regression model the 
regression assumptions from Part A of this chapter must be verified. The Actual by 
Predicted Plot in Figure 65 indicates that the relationship between the response variable 
(coverage) and the regressors is nearly linear. The apparently random cloud of data 


points with a mean of zero in the Residual by Predicted Plot, Figure 67, confirms that the 
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requirements for a zero mean, constant variance, and lack of correlation of the residuals is 
met. Compliance with the normality assumption is shown in Figure 68 by a histogram of 


the residuals overlaid with a normal curve and a normal quantile plot. 
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Figure 67. Residual by Predicted Plot for Coverage Model 
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Figure 68. Histogram and Normal Quantile Plot of Residuals for the Coverage Model 


b. Coverage Factor Interactions 


The Interaction Profile matrix for the coverage model is 16 x 16 cells, too 
large to meaningfully display in this context. The matrix contains 18 cells with 
significant interactions, however, just as in the non-penetrating scenario, the minimum 
and maximum factor setting levels result in only marginal changes to overall coverage. 
Given the minimal effect of the interactions, additional detailed discussion of the 


individual interactions is unwarranted for this study. 
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c. Coverage Regression Tree Analysis 


Just as with the non-penetrating scenario, to determine the number of 
splits that would continue to yield additional explanatory value for Coverage, 20 splits 
were done in rapid succession to record the R-squared value without regard to the content 
or structure of the regression tree. These R-squared values were plotted to find the point 
that the curve began to plateau, indicating diminishing returns from each successive split. 
In this case, 10 splits are warranted. Figure 69 provides a plot of the R-squared values 


against the number of splits. 
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Figure 69. R-Squared vs. Number of Regression Tree Splits 


Figure 70 shows the regression tree for Coverage with 10 splits. The full 
tree and partition graph can be found in Appendix J. The first split indicates that better 
coverage is obtained when the optimization interval is 2.1 hours or greater. In the second 
split, coverage is improved by having one or more U-2s at Base D. The third partition 
demonstrates improved coverage when there are three, the maximum number of RQ-4s, 
at Base C. The fourth split indicates that better coverage is gained by having at least one 
RQ-4 at Base D. The fifth split indicates that better coverage is obtained with the 


maximum number (six) P-3s at Base A. The sixth split indicates that coverage is 
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improved by the stationing at least five RQ-1s at Base C. The seventh split is a return to 
the optimization interval, showing that coverage improves with optimization intervals of 
5.37 hours or more. Note that six hours is the maximum optimization interval for this 
study. The eighth split is on having two or more RQ-Is at Base D. Split nine indicates 
improved coverage by stationing two or more RC-135s at Base D. The final split is on 
RQ-Is at Base A, this time indicating improved coverage with six platforms, the 


maximum number. 


As in the penetrating scenario, optimization interval appears twice and is 
the most significant factor. Interestingly, the optimization interval is the most significant 
factor in both the penetrating and non-penetrating scenarios; however, its level has the 
opposite effect on coverage. In the non-penetrating scenario, as the optimization interval 
increases, coverage goes down. The opposite occurs in the penetrating scenario, where 
coverage improves as the optimization interval goes up. This disparate behavior is likely 
due to addition of mission areas, and consequently, more reachable missions in the 
penetrating scenario. A greater variety of mission area to mission assignment options 
reduces the chance of the most capable platforms going unused due to assignment 


saturation caused by shorter optimization intervals. 


In this regression tree, all types of aircraft are represented; however, Base 
B does not appear. That is not to suggest that Base B is irrelevant, only that it is not a 
contributor to the most significant factors. Recall that Base B is furthest from the COI. 
The presence of all platform types is in contrast to the non-penetrating scenario where U- 


2s were heavily represented and P-3s not at all. 
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Partition for Mean(coverage) 









































































































































































































































RSquare NN Splits 
Included 0.491 270 10 
Excluded 0.266 4 
All Rows 
OPT INT<2.10416667 OPT INT>=2. 10416667 
C RQ-4<3)|C RQ-4>=3 D U-2<1 D U-2>=1 
D RQ-4<1||D RQ-4>=1 A P-3<6| A P-3>=6 
OPT INT<5.3675 OPT INT>=5.3675] |C RQ-1<5]|C RQ-1>=5 
D RQ-1<2 D RQ-1>=2 















































[D RC-135<2||D Rc-135>=2| [A RO-1<6l[A RO-1>=6 





Figure 70. Regression Tree for Coverage 


d. Coverage Contour Plot Analysis 


To explore the relationship between the various types of ISR platforms 
and the total number of platforms, contour plots were generated. The various color filled 
regions of the plot represent the amount of coverage attained as the number of each 
platform and total aircraft vary. Recall from the non-penetrating scenario contour plot 
analysis that the combinatorial nature of the problem and strong interactions of the 
regressors result in a non-smooth contour plot. Also, note that the surface is affected by 
changes in other factors; therefore, trends in the contour plots are more important than 


specific values. 


Figure 71 shows that in order achieve the best coverage, approximately 60 
total aircraft are required with at least 4 U-2s. Once 60 total aircraft are available, the 


addition of more U-2 platforms continues to improve coverage. 
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Contour Plot for Coverage 
Legend 
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Figure 71. Coverage Contour Plot for U-2 vs. Total Aircraft 


The relationship between the number of RQ-4s and total aircraft is 
illustrated in Figure 72. The total number of aircraft necessary to achieve the best levels 
of coverage is lower than for the U-2 once there are four or more RQ-4s in the 
simulation. As with the U-2, coverage continues to improve with the addition of more 


RQ-4 airframes. 
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Figure 72. Coverage Contour Plot for RQ-4 vs. Total Aircraft 


Figure 73 demonstrates the value of the RC-135 in relation to total 
aircraft. The maximum levels of coverage appear to be achieved when there are at least 
55 total aircraft and 3 or more RC-135s. The RC-135 continues to maximize coverage as 


its quantity and the total number of aircraft rise. 
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Contour Plot for Coverage 
Legend 
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Figure 73. Coverage Contour Plot for RC-135 vs. Total Aircraft 


Figure 74 displays the relationship between the quantity of P-3 airframes 
and the total number of aircraft. The best levels of coverage are reached when there are 
more than 10 P-3s and 60 or more total aircraft. Beyond these levels, coverage continues 
to improve. 
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Figure 74. Coverage Contour Plot for P-3 vs. Total Aircraft 


Figure 75 shows that the best coverage is attained with approximately 60 
total aircraft and more than 12 RQ-1Is. As with all of the other platform types, coverage 


continues to rise with the addition of more airframes. 
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Contour Plot for Coverage 
Legend 





Mean(coverage) 
MM <_ 0.450 
MM <- 0.500 
<= 0.550 
Mm <= 0.600 
Mi <~ 0.650 
MM <- 0.700 
M@® 5 0.700 


RQ-1 

















Sod 
10 20 30 40 50 60 70 80 90 
TOTAL ACFT 








Figure 75. Coverage Contour Plot for RQ-1 vs. Total Aircraft 
4. Coverage by Type 


The Coverage by Type MOE is a subset of total coverage. Figure 76 displays the 
amount of coverage by each platform type. Recall that mean coverage for the non- 
penetrating scenario was 59.2%. Of this total amount of coverage the RQ-4 is the largest 
contributor at 52.3% of the total, far exceeding the next closest contributor, the U-2 at 
19.5%. In the penetrating scenario, these two platforms account for nearly 72% of the 
total coverage. Again, this is not unexpected given the superior sensor packages carried 
by these airframes. The ability of the RQ-4 to remain on station, along with its highly 
capable sensor package, clearly demonstrates its value. The ability to use mission areas 
in country appears to have boosted the P-3s contribution in the penetrating scenario. The 
utility of the RQ-1 declines slightly in the penetrating scenario, likely due to the 
enhanced collection ability of the other platforms given the addition of the penetrating 
mission areas. The RC-135s total contribution appears severely limited by its single 
sensor capability. Also, as in the non-penetrating scenario, this disadvantage is 
compounded by the fact that there are fewer SIGINT targets in the scenario than EO/IR 
or SAR targets. 
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Figure 76. Coverage by Platform Type 
5. Attrition 


Attrition is a relatively uninteresting MOE for this analysis. The lack of realistic 
unclassified performance data and parameters for the threat platforms and vulnerability 
data for the ISR platforms resulted in a generic treatment of this data within the 
simulation. The P, for all ISR platforms at the minimum and maximum surface-to-air 
missile range was set at 0.6 and 0.5, respectively. Consequently, the ISR platforms were 
attrited in exact proportion to their presence in the scenario (See Figure 77). Of note, 
4,139,948 aircraft sorties were generated and flown in this simulation, 2% or 94,390 were 
killed. Attrition for studies conducted in a classified environment with real world 


parameters would likely differ significantly from the results found in this experiment. 
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Figure 77. Proportional Attrition of ISR Platforms 


6. Comparison of Non-Penetrating vs. Penetrating Results 


Several significant similarities and differences between the output from the non- 
penetrating and penetrating scenarios were identified. These findings and an explanation 


of the underlying factors is provided. 


a. Coverage 


As expected, the coverage for the penetrating scenario exceeds that of the 
non-penetrating scenario (See Figure 78). The non-penetrating scenario resulted in a 
mean coverage of 0.49, with a standard deviation of 0.02, and maximum and minimum 
values of 0.53 and 0.43, respectively. The penetrating scenario yielded a mean coverage 
of 0.59, with a standard deviation of 0.06, and maximum and minimum values of 0.79 
and 0.40, respectively. The reduced coverage in the non-penetrating scenario is due to 
the physical inability of the sensors to range the targets. It must be noted that coverage 
for the penetrating scenario could potentially be much lower if the attrition rate was 


increased. 


While the penetrating scenario provides 10% more mean coverage, its 


results contain 3 times the variability. In fact, the lowest coverage value is actually found 
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in the results of the penetrating scenario. This variability is due to both the stochastic 
nature of the penetrating scenario and relative increased complexity of the approximate 


dynamic programming approach due to the addition of more mission areas. 
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Figure 78. Penetrating vs. Non-Penetrating Coverage 
c. Coverage by Type 


A comparison of the coverage by platform type for each scenario is shown 
in Figure 79. The RQ-4 provides the greatest amount of coverage in both scenarios; 
however it is nearly matched by the U-2 in the non-penetrating scenario. This disparity is 
likely due to the RQ-4s ability to remain on-station, particularly in the penetrating 
scenario where additional transit time is required to reach the mission areas. The RQ-4 
and U-2 superior coverage is not unexpected as both airframes carry all three of the 
sensor types considered for this study. The substantial demand for the three sensor types 
impacts the value of the most capable platforms. The combined coverage of the 
remaining platforms, the RQ-1, the P-3, and the RC-135, is approximately 25% of the 
total in both scenarios. None of these aircraft has more than two sensor types, lowering 
their ability to contribute significantly to coverage. Of note, the P-3 provides almost no 
value in the non-penetrating scenario, likely due to its relatively short on station time 


when compared to the RQ-1 which has a similar sensor package. 
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Figure 79. Non-Penetrating vs. Penetrating Coverage by Platform Type 
b. Factor Comparison 


Figure 80 illustrates the opposite effect demonstrated by the optimization 
interval factor with respect to coverage. In the non-penetrating scenario, shorter 
optimization intervals are better, but in the penetrating scenario longer optimization 
intervals yield improved coverage. This disparity is explained by the increase in mission 
areas and reachable missions available in the penetrating scenario. With more mission 
areas and missions included in each optimization, there is less chance that the more 
capable airframes will be underutilized due to assignment saturation. Recall from 
Chapter II that all open missions and available airframes are matched at each 
optimization event. When there are a greater number of mission areas and missions 
available, as in the penetrating scenario, there is less chance that the better platforms will 


be assigned to low target density areas or not used at all. 
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Figure 80. Bivariate Fit of Coverage by Optimization Interval for the Non- 
Penetrating and Penetrating Scenarios 


The U-2 and the RQ-4 show up as significant factors in both scenarios. 
An examination of the regression trees from each scenario, Figures 60 (page 90) and 70 
(page 99), reveals that the dataset is partitioned on U-2 as a factor before the RQ-4 even 
though the RQ-4 provides a greater percentage of the overall coverage in both scenario. 
This is due to the RQ-4s long endurance. The long dwell time of the RQ-4 means that it 
is involved in fewer optimization events and therefore does not contribute to the 


variability of the coverage in the same way as the U-2. 
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VI. FINDINGS AND RECOMMENDATIONS 


A. SIGNIFICANT FINDINGS 


JDAFS was exercised to determine its viability for use in future studies. In doing 


SO, scenarios were constructed and simulations run based on Nearly Orthogonal Latin 


Hypercube design of experiment methodology. Based on these experiments, data 


analysis was conducted on the resultant simulation output and issues with the JDAFS 


model were identified for enhancement or correction. The significant findings from this 


research are summarized below. 


2. 


Data Analysis Issues 


Optimization is the most significant factor in both the scenarios, but has 
the opposite effect. In the non-penetrating scenario, shorter optimization 
intervals are better, but in the penetrating scenario longer optimization 
intervals yield improved coverage. Increased assignment options in the 
penetrating scenario causes this disparate behavior by the optimization 
interval factor. 


The U-2 is the most important platform factor in explaining the variability 
in both scenarios. 


The RQ-4 results in the majority of mission coverage for both scenarios 
due to its ability to remain on station. 


Coverage is not necessarily improved by adding more ISR assets in the 
non-penetrating scenario. 


Coverage is improved by the addition of assets in the penetrating scenario. 
Coverage would likely eventually also plateau with diminishing returns 
from the addition of more aircraft in this scenario. The maximum number 
of airframes in the scenario was not sufficient to explore this behavior. 


Penetrating ISR platforms average 20% more mean coverage than non- 
penetrating platforms. 


Issues Identified and Corrected During Preliminary Testing 


The exercise of the JOAFS model prior to the commencement of production runs 


identified a number of bugs and deficiencies. These discrepancies were all promptly 


addressed and corrected by the model developers. The following is a summary of the 


issues and modifications made to JDAFS. 
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a. JDAFS Architectural Issues 


Initial attempts to run JDAFS in a cluster environment were unsuccessful. 
JDAFS was originally written to be run from a GUI. The basic window object for a GUI 
in the Java programming language is called a "JFrame". The model initialization code for 
JDAFS was placed inside a JFrame constructor. Thus, even if no window was opened, a 
JFrame needed to be constructed to run JDAFS. In order to construct a JFrame, Java 
polls the environment to determine all of the characteristics of the GUI environment. 
Since there wasn't any such environment when running JDAFS from the command line in 
a cluster environment, this caused a "Null pointer exception", i.e., a reference to 
something which doesn't exist. This is a terminal error for a Java program. The model 
developers removed the model initialization code from JDAFS and built a completely 


new installer that ensured cluster compatibility. 


Aircraft routing was initially done on a straight line path from a base to its 
associated mission areas. To prevent the overflight of denied areas, it was necessary to 
implement a waypoint routing capability. The “route” table was added to give the ability 
to specify ingress and egress routes between bases and mission areas or between two 


mission areas. 


b. JDAFS Bugs Identified and Corrected 


JDAFS uses the Bugzilla open-source defect tracking system to allow 
users to report issues discovered when using the simulation. Bugzilla also provides a 
means for the JDAFS developers to communicate the status of correcting any 
deficiencies. All of the bugs identified in this research were promptly addressed and 
corrected, often in less than 24 hours. Table 16 list the bugs identified, their symptoms, 


and the necessary corrections made in JDAFS. 
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Symptom 


Problem 


Correction 





JDAFS simulation failed to 
run to completion. An 
Illegal State Transition 
Exception error was thrown 
in JAVA. 


If a mission platform was 
killed on the ground, its 
pending events were not 
cancelled. Since a dead 
platform is unable to 
execute its mission, JAVA 
throws the Illegal State 
Transition Exception and 
the simulation cannot 
continue. 


The JDAFS JAVA code 
was amended to ensure all 
pending events are 
cancelled for killed mission 
platforms. 





JDAFS Simulation 
optimization event took 
successively longer to 
complete, eventually 
resulting in a failure to run 
to completion. 


JDAFS was not calling 
deleteLpQ) following 
optimization events. This 
resulted in a memory leak 
in Ipsolve. As the 
simulation progressed, there 
were insufficient memory 
resources for the program to 
continue. 


A call to deleteLp() was 
added in the 3D CVO 
process in JAVA to ensure 
that memory resources were 
freed prior to each 
optimization event. 





Unusually high coverage 
values were reported 
following simulation runs. 





Table 16. 


Platforms were receiving 
credit for coverage of 
targets that were out of 
range or for which LOS did 








JDAFS coverage reports 
were corrected to ensure 
coverage credit was only 
received when platforms 


not exist. actually covered a mission. 
JDAFS Bugs Identified and Corrected 


B. RECOMMENDATIONS FOR FURTHER JDAFS IMPROVEMENT 


JDAFS is a powerful research tool; however, a number of modifications are 


warranted that will improve the usability, functionality, and credibility of the model for 


future studies. The following changes and improvements are recommended: 


e Allow ISR platforms to receive credit for collection on targets within 
sensor range during ingress and egress to mission areas. 


° Create probability distributions to represent current and future force 
sensors from certified data to accurately represent the acquisition of 
targets by sensors in the model. 


e Implement revisit valuation schemes for missions to more credibly 
represent ISR mission characteristics. 
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Addition of joint assets (platforms, sensors, munitions, etc.) in the JDAFS 
database for reuse in joint studies. 


Further develop the functionality of JDAFS to more credibly represent 
Joint Sensors to include satellite assets. 


Provide a more robust representation of unmanned aerial vehicles with 
weapons and sensors that allow force-on-force analysis of differing 
platform mixes. 


Develop an integrated design of experiments interface that enables a quick 
determination of alternatives. 


Incorporate the InputGenerator and OutputGenerator scripts into the 
JDAFS program. The integration of these tools will eliminate the need for 
the pre and post-processing of data and eliminate the need to transfer 
excessively large files. 


Develop a simple, well documented JDAFS interface to allow an analyst 
to easily enter the data for the various input factors. 


Develop a comprehensive JOAFS Users/Analyst Manual. 


C. RECOMMENDATIONS FOR FUTURE JDAFS STUDIES 


The research conducted for this thesis exercised the JDAFS model and 


demonstrated its utility as a simulation and data farming tool. The following areas should 


be considered for additional research using JDAFS: 


Exercise data farming concepts as additional development is done to the 
JDAFS model. 


Explore the use of JDAFS to study UAV acquisition of moving or pop-up 
targets. 


Conduct analysis using real world terrain and order of battle to explore 
multiple operational factors. 


Explore the use of JDAFS as an acquisition and capabilities requirements 
tool. 


Use JDAFS to explore the trade-offs associated with manned vs. 
unmanned ISR platforms. 


Evaluate the utility of JDAFS as a screening tool for high resolution 
modeling and simulation. 


Allow cueing and flexible time windows to further enhance scenario 
realism. 


114 


APPENDIX A. NON-PENETRATING SCENARIO BASE TO 
MISSION AREA DISTANCE TABLE 





Pp t—“‘C‘’SC*é@“ASECSS 

(ae ae a 
AREA 

328000 | 1195861 | 1250502 | 1620154 
356629 | 1061925 | 1110502 | 1480154 
424834 | 939140 | 980502 | 1350154 
569745 | 936862] 861140 | 1205242 
714656 | 956785 | 750824 | 1060331 
859568 | 997578] 654149 | 915420 
1004479 | 1056827 | 578000 | 770509 
1149390 | 1199525 | 654149 | 625629 
1294301 | 1342698 | 750824 | 480767 
1439212 | 1486207 861140 | 335948 
1584124 | 1629964 | 980502] 191269 
1714124 | 1759964 | 1092742 | 187840 
1854124 | 1899964 | 1217573 | 267739 
356629 | 1331046 | 1390502 | 1760154 
MIN 328000 | 936862] 578000 | 187840 
MAX 1854124 | 1899964 | 1390502 | 1760154 
MEAN 974987 | 1271025 | 938075 | 875094 
MEDIAN 932023 | 1197693 | 920821 | 842965 
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Non-Penetrating Scenario: Base to Mission Area Distance ( in meters) Table (The 
additional transit required due to waypoint routing to prevent COI overflight is included 


in the distance calculations for this table). 
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APPENDIX B. PENETRATING SCENARIO BASE TO MISSION 
AREA DISTANCE TABLE 





| C“(‘éiBASES Cd 

| “awe | * | & | © | ? | 
AREA 

328000 | 1195861 | 1217573 | 1238258 
356629 | 1061925 | 1092742 | 1223472 
424834 | 939140 | 980502 | 1224003 
566667 | 936862 | 861140 | 1099573 
709744 | 956785 | 750824} 980784 
853441 | 997578 | 654149} 869949 
997489 | 1056827 | 578000} 770509 
1082618 | 1199525 | 654149 |} 625629 
1179465 | 1342698 | 750824 | 480767 
1285384 | 1486207 | 861140} 335948 
1398315 | 1629964 | 980502} 191269 
1379124 | 1703668 | 1092742 | 187840 
1372000 | 1790219 | 1217573 | 267739 
356629 | 1331046 | 1345394 | 1268418 
450000 | 1234909 | 1170470 | 1118034 
672301 | 1504024 | 1329018 | 1000744 
672301 | 1134411 | 912956 | 912079 
700000 | 1346291 | 1110180 | 873212 
939415 | 1767767 | 1500000 | 921954 
863134 | 1553222 | 1250000 | 782624 
863134 | 1312440 | 950000} 701783 
939415 | 1133578 | 700000} 728011 
1000000 | 1523975 | 1110180 | 585235 
1082953 | 1727104 | 1329018 | 638661 
1082953 | 1416929 | 912956 |} 488148 
1250000 | 1698529 | 1170470 | 360555 
MIN 328000 | 936862 | 578000} 187840 
MAX 1398315 | 1790219 | 1500000 | 1268418 
MEAN 877152 | 1345442 | 1018558 | 764431 
MEDIAN 901274 | 1336872 | 1036622 | 776566 


Penetrating Scenario: Base to Mission Area Distance (in meters) Table 
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APPENDIX C. BASE ‘A’ TRANSIT AND ON-STATION 
STATISTICS 


NON-PENETRATING SCENARIO 


MISSION RQ-4 RQ-1 RC-135 U-2 
AREA | TRANS. ON-STA] TRANS. ON-STA rae oon STA | TRANS. ON-STA] TRANS. ON-STA 


1.041799 4.427646 1.55356 0.814271 0.795981 
1.13273 ; 4.814102 : 1.689159 - 0.885342 : 0.865456 
1.349365 ; 5.734801 2 2.012211 9. 1.054664 9. 1.030976 
1.809634 5 7.690946 : 2.698577 9. 1.414411 9. 1.382642 
2.269904 3 9.64709 : 3.384944 8. 1.774158 9. 1.734308 
2.730173 : 11.60323 : 4.07131 2.133904 8. 2.085975 
3.190442 : 13.55938 : 4.757677 7. 2.493651 8. 2.437641 
3.650711 é 15.51552 8. 5.444043 6. 2.853398 8. 2.789308 
4.110981 : 17.47167 6. 6.13041 rs 3.213144 7. 3.140974 
19.42781 4. 6.816776 5. : . 3.49264 

21.38396 2. 7.503143 4. ; ; 3.844307 

23.13882 0. 8.118883 3. j : 4.159787 

24 ‘ : : ; 4.499535 

4.814102 : : - : : 0.865456 


PENETRATING SCENARIO 


a Sc, 
MISSION RQ-4 RQ-1 RC-135 U-2 
AREA | TRANS. ON-STA| TRANS. ON-STA TRANS. “ON STA | TRANS. ON-STA| TRANS. _ON-STA 


1.041799 4.427646 1.55356 0.814271 0.795981 
1.13273 ; 4.814102 ; 1.689159 - 0.885342 F 0.865456 
1.349365 ; 5.734801 : 2.012211 9. 1.054664 9. 1.030976 
1.799856 : 7.64939 ; 2.683996 9. 1.406769 9. 1.375171 
2.254302 ; 9.580782 ; 3.361678 8. 1.761963 9. 1.722388 
2.710713 : 11.52053 : 4.042291 7. 2.118694 8. 2.071106 
3.168241 : 13.46502 ; 4.724569 7. 2.476298 8. 2.420678 
3.438628 : 14.61417 9. 5.127779 6. 2.687634 8. 2.627267 
3.746235 ‘ 15.9215 : 5.586491 6. 2.928059 8. 2.862292 
4.082657 ‘ 17.35129 6. 6.088173 5. 3.191007 7. 3.119333 
4.44135 : 18.87574 5. 6.623066 5. 3.471361 7. 3.393391 
4.380398 . 18.61669 . 6.532172 5. 3.423721 7. 3.346821 
4.357769 ‘ 18.52052 5. 6.498427 5. 3.406034 7. 3.329531 
1.13273 : 4.814102 : 1.689159 - 0.885342 F 0.865456 
1.429297 E 6.074514 : 2.131408 9. 1.11714 5 1.092047 
2.135372 : 9.075331 : 3.184327 8. 1.669008 9. 1.63152 
2.135372 : 9.075331 : 3.184327 8. 1.669008 9. 1.63152 
2.223352 : 9.449244 : 3.315524 8. 1.737773 9. 1.698741 
2.983784 5 12.68108 : 4.449503 7. 2.332127 8. 2.279745 
2.7415 B 11.65137 i 4.088202 7. 2.142758 8. 2.094629 
2.7415 ‘ 11.65137 : 4.088202 7. 2.142758 8. 2.094629 
2.983784 5 12.68108 : 4.449503 7. 2.332127 8. 2.279745 
3.176216 : 13.49892 ; 4.736463 7. 2.482532 8. 2.426772 
3.439694 : 14.6187 9. 5.129369 6. 2.688467 8. 2.628081 
3.439694 : 14.6187 9. 5.129369 6. 2.688467 8. 2.628081 
3.970271 ; 16.87365 : 5.920579 6. 3.103165 7. 3.033465 
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APPENDIX D. BASE ‘B’ TRANSIT AND ON-STATION 
STATISTICS 


NON-PENETRATING SCENARIO 


— RQ-4 RQ-1 RC-135 U-2 
AREA | TRANS. ON-STA] TRANS. ON-STA TRANS. “on. — TRANS. ON-STA] TRANS. ON-STA 


3.798314 16.14283 2.968764 2.902083 
3.372903 A 14.33484 9. ; : 2.636262 8. 2.577049 
2.982912 , 12.67738 : . : 2.331445 8. 2.279079 
2.975678 : 12.64663 : : : 2.325791 8. 2.273552 
3.038955 ; 12.91556 : . : 2.375249 8. 2.321899 
3.168523 ; 13.46622 : : P 2.476519 8. 2.420894 
3.356712 : 14.26603 9. . ; 2.623608 8. 2.564679 
3.809952 : 16.1923 7. F : 2.977861 8. 2.910975 
4.264698 ‘ 18.12497 5. : : 3.33329 ‘ 3.258421 
4.720515 5 20.06219 ; ‘ ; 3.689557 7. 3.606686 

5.17712 : 22.00276 1. : ; 4.046439 6. 3.955552 
5.590028 ; 23.75762 0. ; : 4.369168 6. 4.271033 
6.034698 R 24 : i 4.716723 6. 4.610781 
4.227691 ‘ 17.96769 6. : j 3.304365 7. 3.230146 


PENETRATING SCENARIO 


ae 
MISSION RQ-4 RQ-1 RC-135 U-2 
aie TRANS. ON-STA| TRANS. ON-STA TRANS. “ON: STA | TRANS. ON-STA]| TRANS. _ON-STA 


3.798314 16.14283 5.664153 2.968764 8.031236} 2.902083 
3.372903 A 14.33484 9. 5.029767 6. 2.636262 8.363738] 2.577049 
2.982912 , 12.67738 : 4.448202 7. 2.331445 8.668555] 2.279079 
2.975678 : 12.64663 ‘ 4.437415 7. 2.325791 8.674209] 2.273552 
3.038955 ; 12.91556 : 4.531776 7. 2.375249 8.624751] 2.321899 
3.168523 f 13.46622 : 4.72499 2.476519 8.523481] 2.420894 
3.356712 ; 14.26603 9. 5.005624 6. 2.623608 8.376392] 2.564679 
3.809952 3 16.1923 7. 5.681508 6. 2.977861 8.022139] 2.910975 
4.264698 . 18.12497 5. 6.359638 5. 3.33329 7.66671] 3.258421 
4.720515 5 20.06219 ‘ 7.039365 4. 3.689557 7.310443] 3.606686 
5.17712 F 22.00276 1. 7.720266 4. 4.046439 6.953561} 3.955552 
5.411218 ; 22.99768 1. 8.06936 5 4.22941 6.77059] 4.134414 
5.686123 j 24 8.479306 3. 4.444276 6.555724] 4.344454 
4.227691 : 17.96769 6. 6.304451 5. 3.304365 7.695635} 3.230146 
3.922338 3 16.66994 7. 5.849101 6. 3.065701 7.934299] 2.996843 
20.3027 3. 7.123754 4. 3.733788 7.266212] 3.649923 

15.31332 8. 5.373095 6. 2.816212 8.183788] 2.752957 

18.17348 5. 6.376659 5. 3.342211 7.657789] 3.267142 

: F 23.86294 0. 8.372963 3. 4.388539 6.611461] 4.289968 
4.933371 ‘ 20.96683 3. 7.356781 4. 3.855925 7.144075] 3.769317 
4.168595 A 17.71653 6. 6.216326 5. 3.258176 7.741824] 3.184994 
3.60049 : 15.30208 8. 5.369152 6. 2.814145 8.185855] 2.750937 
4.840475 : 20.57202 3. 7.218252 4. 3.783317 7.216683] 3.69834 
5.485656 ; 23.31404 0. 8.180364 3. 4.287591 6. 4.191288 
4.500473 : 19.12701 4. 6.711232 5. 3.517572 7. 3.438564 
5.394895 if 22.9283 1. 8.045019 3. 4.216653 6. 4.121942 
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APPENDIX E. BASE ‘C’ TRANSIT AND ON-STATION 
STATISTICS 


NON- ——e—e—ero. SCENARIO 


— RQ-4 RQ-1 = 3 RC-135 U-2 
AREA | TRANS. ON-STA] TRANS. ON-STA | TRANS. ~ =i) TRANS. ON-STA] TRANS. ON-STA 


3.971865 16.88043 5.922956 3.104411 7.895589} 3.034683 
3.527194 : 14.99058 9. 5.259851 6. 2.756857 8.243143] 2.694935 
3.114286 ? 13.23572 ‘ 4.644111 7. 2.434128 8.565872] 2.379455 
2.735166 ; 11.62445 : 4.078756 7. 2.137807 8.862193] 2.089789 

2.38478 ; 10.13532 ; 3.556251 8. 1.863945 9.136055] 1.822079 

2.07772 : 8.830309 : 3.098354 8. 1.623947 9.376053} 1.587471 
1.835853 ; 7.802376 : 2.737676 9. 1.434904 9.565096] 1.402674 

2.07772 ; 8.830309 ; 3.098354 8. 1.623947 9.376053} 1.587471 

2.38478 ; 10.13532 : 3.556251 8. 1.863945 9.136055] 1.822079 
2.735166 : 11.62445 : 4.078756 7. 2.137807 8. 2.089789 
3.114286 : 13.23572 : 4.644111 7. 2.434128 8. 2.379455 
3.470784 f 14.75083 i 5.17573 - 2.712766 8. 2.651835 
3.867275 ‘ 16.43592 7. 5.76699 i 3.022664 7. 2.954772 
4.416535 ; 18.77027 5. 6.586061 5. 3.451966 7. 3.374431 


PENETRATING SCENARIO 


eS 
MISSION RQ-4 RQ-1 P-3 RC-135 U-2 
AREA | TRANS. ON-STA| TRANS. ON-STA| TRANS. ON-STA| TRANS. ON-STA] TRANS. _ON-STA 


3.867275 16.43592 5.76699 3.022664 2.954772 
3.470784 : 14.75083 : 5.17573 : 2.712766 8. 2.651835 
3.114286 : 13.23572 : 4.644111 7. 2.434128 8. 2.379455 
2.735166 ; 11.62445 : 4.078756 7. 2.137807 8. 2.089789 

2.38478 5 10.13532 ; 3.556251 8. 1.863945 9. 1.822079 

2.07772 ; 8.830309 2 3.098354 8. 1.623947 9. 1.587471 
1.835853 : 7.802376 : 2.737676 9. 1.434904 9. 1.402674 

2.07772 . 8.830309 ; 3.098354 8. 1.623947 9. 1.587471 

2.38478 : 10.13532 . 3.556251 8. 1.863945 9. 1.822079 
2.735166 ; 11.62445 : 4.078756 7. 2.137807 8. 2.089789 
3.114286 : 13.23572 : 4.644111 7. 2.434128 8. 2.379455 
3.470784 ! 14.75083 i 5.17573 - 2.712766 8. 2.651835 
3.867275 ; 16.43592 7. 5.76699 3 3.022664 7. 2.954772 
4.273261 5 18.16136 5. 6.372407 5. 3.339983 7. 3.264964 
3.717666 : 15.80008 8. 5.543888 6. 2.90573 , 2.840464 
4.221248 ; 17.9403 6. 6.294843 5. 3.299329 7. 3.225223 
2.899745 : 12.32392 . 4.324181 7. 2.266442 8. 2.215535 
3.526173 : 14.98623 9. 5.258327 6. 2.756058 8. 2.694154 
4.764325 ; 20.24838 : 7.104695 4. 3.723798 7. 3.640158 
3.970271 : 16.87365 : 5.920579 6. 3.103165 7. 3.033465 
3.017406 : 12.82397 d 4.49964 2.358406 8. 2.305434 
2.223352 : 9.449244 ; 3.315524 8. 1.737773 9. 1.698741 
3.526173 ‘ 14.98623 9. 5.258327 6. 2.756058 8. 2.694154 
4.221248 5 ; a 6.294843 5. 3.299329 7. 3.225223 
2.899745 : : : ; : 2.266442 8. 2.215535 
3.717666 j ; A ; ; 2.90573 : 2.840464 





123 


THIS PAGE INTENTIONALLY LEFT BLANK 


124 


APPENDIX F. BASE ‘D’ TRANSIT AND ON-STATION 
STATISTICS 


NON-PENETRATING SCENARIO 


— RQ-4 RQ-1 RC-135 U-2 
AREA | TRANS. ON-STA] TRANS. ON-STA Tass “on = TRANS. ON-STA] TRANS. ON-STA 


5.145959 21.87032 7.673798 4.022084 6.977916] 3.931744 
4.701288 i 19.98048 4. 7.010693 4. 3.674529 7.325471] 3.591996 

4.28838 5 18.22562 5. 6.394953 5. 3.3518 7.6482] 3.276515 
3.828111 : 16.26947 7. 5.708587 6. 2.992053 8.007947] 2.924849 
3.367842 ; 14.31333 9. 5.02222 5 2.632307 8.367693] 2.573182 
2.907572 : 12.35718 : 4.335854 7. 2.27256 8.72744] 2.221516 
2.447303 R 10.40104 : 3.649487 8. 1.912813 9.087187] 1.86985 
1.987132 ; 8.445312 . 2.963267 9. 1.553143 9.446857] 1.518258 

1.52702 : 6.489837 : 2.277136 9. 1.19352 9.80648] 1.166712 
1.067044 : 4.534939 : 1.591207 F 0.834002 10.166] 0.81527 
0.607513 : 2.581931 : 0.905941 ; 0.474833 10.52517] 0.464167 
0.596622 F 2.535642 . 0.889699 : 0.46632 10.53368] 0.455846 
0.850396 2 3.614183 : 1.268134 5 0.66467 10.33533} 0.649741 
5.590629 : 23.76017 0. 8.336903 3. 4.369638 6. 4.271492 


PENETRATING SCENARIO 


eee 
MISSION RQ-4 RQ-1 RC-135 U-2 
wie TRANS. ON-STA| TRANS. ON-STA TRANS. “ON: STA | TRANS. ON-STA]| TRANS. _ON-STA 


3.932977 16.71515 5.864966 3.074017 7.925983] 3.004971 
3.886012 j 16.51555 7. 5.794931 6. 3.037309 7.962691] 2.969088 
3.887699 : 16.52272 7. 5.797446 6. 3.038628 7.961372] 2.970377 
3.492483 : 14.84305 9. 5.208088 6. 2.729726 8.270274] 2.668414 
3.115182 : 13.23952 : 4.645447 7. 2.434828 8.565172] 2.380139 
2.763146 ; 11.74337 ; 4.120481 7. 2.159676 8.840324] 2.111168 
2.447303 u 10.40104 ‘ 3.649487 8. 1.912813 9.087187] 1.86985 
1.987132 ; 8.445312 : 2.963267 9. 1.553143 9.446857] 1.518258 
1.52702 ‘ 6.489837 : 2.277136 9. 1.19352 9.80648] 1.166712 
1.067044 : 4.534939 : 1.591207 5 0.834002 10.166] 0.81527 
0.607513 : 2.581931 : 0.905941 , 0.474833 10.52517] 0.464167 
0.596622 F 2.535642 . 0.889699 ; 0.46632 10.53368}] 0.455846 
0.850396 fl 3.614183 2 1.268134 f 0.66467 10.33533} 0.649741 
4.02877 ; 17.12227 6. 6.007815 5. 3.148888 7.851112] 3.078161 
3.551118 : 15.09225 8. 5.295527 6. 2.775555 8.224445] 2.713214 
13.50896 : 4.739986 7. 2.484379 8.515621] 2.428577 

12.31208 . 4.320028 7. 2.264265 8.735735] 2.213408 

11.78743 : 4.135939 7. 2.167778 8.832222] 2.119088 

: $ 12.44539 : 4.366803 7. 2.288782 8.711218] 2.237373 
2.485783 f 10.56458 : 3.706869 8. 1.942889 9.057111] 1.89925 
2.229016 : 9.473319 . 3.323971 8. 1.7422 9.2578] 1.703069 
2.312321 : 9.827362 : 3.448197 8. 1.807311 9.192689] 1.766717 
1.858833 ; 7.90004 : 2.771944 9. 1.452865 9.547135] 1.420232 
2.028526 : 8.621235 . 3.024995 8. 1.585497 9.414503] 1.549885 
1.550462 ; 6.589464 : 2.312093 9. 1.211842 9. 1.184623 
1.145201 é 4.867105 f 1.707756 R ; ; 0.874985 
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APPENDIX G. NON-PENETRATING COVERAGE MODEL 


Summary of Fit 


RSquare 0.780377 
RSquare Adj 0.763686 
Root Mean Square Error 0.008541 
Mean of Response 0.485819 
Observations (or Sum Wgts) 270 

Analysis of Variance 
Source DF Sumof Squares Mean Square F Ratio 
Model 19 0.06479516 0.003410 46.7534 
Error 250 0.01823541 0.000073 Prob>F 
C. Total 269 0.08303057 <.0001 * 

Parameter Estimates 
Term Estimate StdError tRatio Prob>|t| 
Intercept 0.4762323 0.002843 167.51 <.0001* 
A RQ-4 -0.002767 0.000534 -5.18  <.0001* 
A RQ-1 -0.000885 0.000285 -3.10 0.0021* 
A RC-135 0.0039884 0.000535 7.45 <.0001* 
A U-2 0.0056309 0.000536 10.50 <.0001* 
B RQ-4 -0.002767 0.000541 -5.12  <.0001* 
B RC-135 -0.000835 0.000538 -1.55 0.1219 
BU-2 0.0035851 0.000534 6.71 <.0001* 
Cc RQ-4 -0.002511 0.000531 -4.73 <.0001* 
CU-2 0.0058404 0.000531 10.99 <.0001* 
D RQ-4 -0.001957 0.000529 -3.70 0.0003* 
D U-2 0.0074843 0.000532 14.07 <.0001* 
OPT INT -0.004388 0.000361 -12.15 <.0001* 
(A RQ-4-1.52963)*(B RC-135-1.52963) -0.002434 0.000522 -4.66 <.0001* 
(A RC-135-1.52963)*(D RQ-4-1.52963) -0.001769 0.000518 -3.41 0.0007* 
(B RC-135-1.52963)*(D RQ-4-1.52963) -0.001793 0.000535 -3.35 0.0009* 
(C RQ-4-1.52963)*(D U-2-1.52963) -0.002549 0.000504 -5.05 <.0001* 
(B RQ-4-1.52963)*(B RQ-4-1.52963) 0.001986 0.000574 3.46 0.0006* 
(C U-2-1.52963)*(C U-2-1.52963) -0.001624 0.00057 -2.85 0.0047* 
(D RQ-4-1.52963)*(D RQ-4-1.52963) 0.0030399 0.000559 5.44 <.0001* 


Scaled Estimates 
Continuous factors centered by mean, scaled by range/2 























Term Scaled Estimate 
Intercept 0.4823027 
A RQ-4 -0.00415 
A RQ-1 -0.002655 
A RC-135 0.0059826 
A U-2 0.0084464 
B RQ-4 -0.00415 
B RC-135 -0.001253 
BU-2 0.0053776 
Cc RQ-4 -0.003766 
CU-2 0.0087605 
D RQ-4 -0.002936 
D U-2 0.0112265 
OPT INT -0.011061 
(A RQ-4-1.52963)*(B RC-135-1.52963) -0.005477 
(A RC-135-1.52963)*(D RQ-4-1.52963) -0.00398 
(B RC-135-1.52963)*(D RQ-4-1.52963) -0.004035 
(C RQ-4-1.52963)*(D U-2-1.52963) -0.005736 
(B RQ-4-1.52963)*(B RQ-4-1.52963) 0.0044685 
(C U-2-1.52963)*(C U-2-1.52963) -0.003654 
(D RQ-4-1.52963)*(D RQ-4-1.52963) 0.0068398 
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Std Error 
0.001085 
0.000802 
0.000856 
0.000803 
0.000805 
0.000811 
0.000807 
0.000801 
0.000796 
0.000797 
0.000793 
0.000798 

0.00091 
0.001175 
0.001166 
0.001204 
0.001135 
0.001292 
0.001282 
0.001258 


t Ratio 
444.69 
-5.18 
-3.10 
7.45 
10.50 
-5.12 
-1.55 
6.71 
-4.73 
10.99 
-3.70 
14.07 
-12.15 
-4.66 
-3.41 
-3.35 
-5.05 
3.46 
-2.85 
5.44 


Prob>|t| 
0.0000* 
<.0001 * 
0.0021 * 
<.0001 * 
<.0001 * 
<.0001 * 
0.1219 

<.0001 * 
<.0001 * 
<.0001 * 
0.0003* 
<.0001 * 
<.0001 * 
<.0001 * 
0.0007* 
0.0009* 
<.0001 * 
0.0006* 
0.0047* 
<.0001 * 
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APPENDIX H. PENETRATING COVERAGE MODEL 


Summary of Fit 








RSquare 0.709558 

RSquare Adj 0.677153 

Root Mean Square Error 0.031621 

Mean of Response 0.592963 

Observations (or Sum Wgts) 270 

Analysis of Variance 

Source DF Sum of Squares Mean Square F Ratio 

Model 27 0.59115961 0.021895 21.8968 

Error 242 0.24197826 0.001000 Prob>F 

C. Total 269 0.83313787 <.0001* 

Parameter Estimates 

Term Estimate StdError tRatio Prob>|t| 
Intercept 0.3606258 0.011655 30.94 <.0001* 
A RQ-4 0.0054156 0.001988 2.72 0.0069* 
ARQ-1 0.0065338 0.001058 6.17 <.0001* 
AP-3 0.0061525 0.001063 5.79 <.0001* 
AU-2 0.0059377 0.001998 2.97 0.0033* 
BRQ-4 0.0080442 0.00197 4.08 <.0001* 
BP-3 0.003962 0.001046 3.79 0.0002* 
BU-2 0.0064787 0.001974 3.28 0.0012" 
Cc RQ-4 0.0107559 0.001982 5.43 <.0001* 
CP-3 0.0022811 0.001052 2.17 0.0311* 
CcU-2 0.0050889 0.001977 2.57 0.0107* 
D RQ-4 0.0063526 0.001993 3.19 0.0016" 
D RQ-1 0.003488 0.001069 3.26 0.0013* 
D P-3 0.0081541 0.001063 7.67 <.0001* 
D RC-135 0.0071914 0.002009 3.58 0.0004* 
DU-2 0.0136722 0.002008 6.81 <.0001* 
OPT INT 0.0139716 0.001338 10.45 <.0001* 
(A RQ-4-1.52963)*(OPT INT-3.46986) -0.003346 0.001438  -2.33 0.0208" 
(A RQ-1-3.05926)*(D U-2-1.52963) 0.0032962 0.001103 2.99 0.0031* 
(A P-3-3.05926)*(C RQ-4-1.52963) 0.0009105 0.000949 0.96 0.3383 
(A U-2-1.52963)*(D RQ-4-1.52963) -0.004324 0.001949 -2.22 0.0274* 
(B RQ-4-1.52963)*(B U-2-1.52963) -0.003686 0.002221 -1.66 0.0983 
(C RQ-4-1.52963)*(OPT INT-3.46986) -0.003959 0.001359 -2.91 0.0039* 
(C P-3-3.05926)*(C U-2-1.52963) -0.001152 0.001217 -0.95 0.3448 
(D RQ-1-3.05926)*(D RC-135-1.52963)  -0.003878 0.001108  -3.50 0.0005* 
(D P-3-3.05926)*(OPT INT-3.46986) 0.0006726 0.000747 0.90 0.3688 
(B P-3-3.05926)*(B P-3-3.05926) -0.001511 0.000673 -2.25 0.0256* 
(D U-2-1.52963)*(D U-2-1.52963) -0.008468 0.00225 -3.76 0.0002* 


Scaled Estimates 
Continuous factors centered by mean, scaled by range/2 






































Term Scaled Estimate Std Error t Ratio Prob>|t| 
Intercept 0.6080798 0.003295 184.53 <.0001* 
ARQ-4 0.0081234 0.002981 2.72 0.0069* 
ARQ-1 0.0196014 0.003174 6.17 <.0001* 
AP-3 0.0184576 0.003189 5.79 <.0001* 
AU-2 0.0089066 0.002997 2.97 0.0033* 
B RQ-4 0.0120663 0.002954 4.08 <.0001* 
BP-3 0.011886 0.003139 3.79 0.0002" 
BU-2 0.009718 0.002962 3.28 0.0012" 
C RQ-4 0.0161338 0.002973 5.43 <.0001* 
CP-3 0.0068433 0.003156 2.17 0.0311* 
CU-2 0.0076333 0.002966 2.57 0.0107* 
D RQ-4 0.0095289 0.002989 3.19 0.0016* 
D RQ-1 0.010464 0.003207 3.26 0.0013* 
DP-3 0.0244623 0.00319 7.67 <.0001* 
D RC-135 0.010787 0.003013 3.58 0.0004* 
DU-2 0.0205083 0.003012 6.81 <.0001* 
OPT INT 0.0352201 0.003372 10.45 <.0001* 
(A RQ-4-1.52963)*(OPT INT-3.46986) -0.01265 0.005438 -2.33 0.0208" 
(A RQ-1-3.05926)*(D U-2-1.52963) 0.0148331 0.004963 2.99 0.0031* 
(A P-3-3.05926)*(C RQ-4-1.52963) 0.0040974 0.004271 0.96 0.3383 
(A U-2-1.52963)*(D RQ-4-1.52963) -0.009729 0.004384 -2.22 0.0274" 
(B RQ-4-1.52963)*(B U-2-1.52963) -0.008293 0.004998 -1.66 0.0983 
(C RQ-4-1.52963)*(OPT INT-3.46986) -0.014971 0.005138 -2.91 0.0039* 
(C P-3-3.05926)*(C U-2-1.52963) -0.005185 0.005477 -0.95 0.3448 
(D RQ-1-3.05926)*(D RC-135-1.52963) -0.017453 0.004984 -3.50 0.0005* 
(D P-3-3.05926)*(OPT INT-3.46986) 0.0050865 0.00565 0.90 0.3688 
(B P-3-3.05926)*(B P-3-3.05926) -0.013603 0.006058 -2.25 0.0256* 
(D U-2-1.52963)*(D U-2-1.52963) -0.019052 0.005062 -3.76 0.0002" 
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APPENDIX I. REGRESSION TREE FOR NON-PENETRATING 


COVERAGE 
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